Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.24 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2471476 2470812 2026-10-20T13:46:24Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers 2471476 wikitext text/x-wiki 7vxcyioeyto48gshtg570kgsyymk3ar 2471503 2471476 2026-10-20T14:06:26Z Stashbot 7414 taavi: reboot bastions to pick up kernel updates 2471503 wikitext text/x-wiki jrk8h29amjk9v2bkl277r6vmigz9k8v 2471522 2471503 2026-10-20T14:14:32Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-3, tools-k8s-control-9 2471522 wikitext text/x-wiki gan0dfg4u81px75ngudo2b11d3vg0le 2471528 2471522 2026-10-20T14:17:45Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-3, tools-k8s-control-9 2471528 wikitext text/x-wiki pzxh9vsjup9hagadkrataxkcvdm544s 2471529 2471528 2026-10-20T14:18:25Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-2, tools-k8s-control-8 2471529 wikitext text/x-wiki lk7o5fgyk1ga7qy8pvdydlozz55lm8r 2471540 2471529 2026-10-20T14:21:43Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-2, tools-k8s-control-8 2471540 wikitext text/x-wiki ht68lfjszkvqtw11cvp4s29087l4gnj 2471541 2471540 2026-10-20T14:21:54Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-control-7 2471541 wikitext text/x-wiki c8k90744lybqw4jmczpts5gj21q1mx0 2471543 2471541 2026-10-20T14:24:02Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-control-7 2471543 wikitext text/x-wiki kw1iwlsi79li5dxje1b5lqpx3cv885p 2471743 2471543 2026-10-20T19:26:19Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers 2471743 wikitext text/x-wiki gtxvdkmkuwvkra5t2xok1qizpvieboj 0 4108 2471728 2470917 2026-10-20T19:03:23Z ScheduleDeploymentBot 37566 Add [[gerrit:1357368]] to Tuesday, October 20 UTC late backport window 2471728 wikitext text/x-wiki gjas8xjwsalie8m2baw5a6r0dsb2zmg 2471729 2471728 2026-10-20T19:03:32Z ScheduleDeploymentBot 37566 Add [[gerrit:1357370]] to Tuesday, October 20 UTC late backport window 2471729 wikitext text/x-wiki 5sqdqdizhcocw2fpjn8zny8jiqujcts 2471765 2471729 2026-10-20T20:12:53Z ScheduleDeploymentBot 37566 Add [[gerrit:1357389]] to Tuesday, October 20 UTC late backport window 2471765 wikitext text/x-wiki rfwxgcqyr6uhs79bruvf8h59vm9e0sv 498 6726 2471478 2470805 2026-10-20T13:46:31Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers 2471478 wikitext text/x-wiki auw9qj7rvfeseb4m6612gthn6t211bm 2471509 2471478 2026-10-20T14:08:18Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers 2471509 wikitext text/x-wiki iv1d2e1ougctn9hzcoak8qo7vhndh7a 2471512 2471509 2026-10-20T14:10:11Z Stashbot 7414 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-gateway-3, toolsbeta-test-k8s-control-12, toolsbeta-test-k8s-gateway-2, toolsbeta-test-k8s-control-11, toolsbeta-test-k8s-gateway-1, toolsbeta-test-k8s-control-10 2471512 wikitext text/x-wiki qeux765ocjnd4ocw8m8au7020t4puhv 2471520 2471512 2026-10-20T14:13:36Z Stashbot 7414 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-gateway-3, toolsbeta-test-k8s-control-12, toolsbeta-test-k8s-gateway-2, toolsbeta-test-k8s-control-11, toolsbeta-test-k8s-gateway-1, toolsbeta-test-k8s-control-10 2471520 wikitext text/x-wiki djdxq9eio928x9pfdhc8jep4sioqb41 0 7919 2471382 2471381 2026-10-20T11:59:33Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1008.eqiad.wmnet 2471382 wikitext text/x-wiki bolb1uemezh4104z5uz6379p9gz54df 2471383 2471382 2026-10-20T11:59:37Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1009.eqiad.wmnet 2471383 wikitext text/x-wiki mu7pur2yiy4fwj2wje6wslkrg6wqqhh 2471384 2471383 2026-10-20T12:00:09Z Stashbot 7414 root@deploy1003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync 2471384 wikitext text/x-wiki jznoizgzrg014c2qhexs25drds7hbcs 2471385 2471384 2026-10-20T12:00:13Z Stashbot 7414 root@deploy1003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync 2471385 wikitext text/x-wiki tpfc2tjbdpke1x0w7ckn0t7n8kro9y7 2471386 2471385 2026-10-20T12:02:53Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1009.eqiad.wmnet 2471386 wikitext text/x-wiki 40fpl52knqpce1id7dsbyedbcl3sdcn 2471387 2471386 2026-10-20T12:02:57Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1012.eqiad.wmnet 2471387 wikitext text/x-wiki gzz8fu67myyus8dktul4auh4np72zzo 2471388 2471387 2026-10-20T12:03:42Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-api-ext-ro in eqiad: T440896 2471388 wikitext text/x-wiki 4nutzhh1um2iu4gtp1kjazeehtxmneh 2471389 2471388 2026-10-20T12:04:50Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1002.eqiad.wmnet 2471389 wikitext text/x-wiki a7lt2v41rl1ktomw06bp2rzx3ib0v06 2471391 2471389 2026-10-20T12:06:42Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T441633)', diff saved to https://phabricator.wikimedia.org/P96879 and previous config saved to /var/cache/conftool/dbconfig/20261020-120640-ladsgroup.json 2471391 wikitext text/x-wiki 5dl05cqaxxx0waa2whecbea2ly1l0gl 2471393 2471391 2026-10-20T12:07:01Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2217.codfw.wmnet with reason: Maintenance 2471393 wikitext text/x-wiki r69vi2bmkdxhluwc3bd2inl5k3qhi8n 2471394 2471393 2026-10-20T12:07:06Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96880 and previous config saved to /var/cache/conftool/dbconfig/20261020-120701-ladsgroup.json 2471394 wikitext text/x-wiki dzf3893ch8bzblnvjbp1xseiirdtmn6 2471395 2471394 2026-10-20T12:07:48Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1012.eqiad.wmnet 2471395 wikitext text/x-wiki msisjan4qvrs59o3opz9f60sh4dse8c 2471396 2471395 2026-10-20T12:07:52Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2008.codfw.wmnet 2471396 wikitext text/x-wiki tgfeik8iytkku7pngkx7x33230b3byv 2471398 2471396 2026-10-20T12:10:56Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1024.eqiad.wmnet with reason: Reboot 2471398 wikitext text/x-wiki 4kk2y0ddmm91eiz2j7pqzm88622o28c 2471399 2471398 2026-10-20T12:11:01Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2008.codfw.wmnet 2471399 wikitext text/x-wiki lxeblhw40bei85d739v3jx99dalnk4b 2471400 2471399 2026-10-20T12:11:05Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2009.codfw.wmnet 2471400 wikitext text/x-wiki 3ciey02bjdp2b9i3zufuu416v8qxw2j 2471401 2471400 2026-10-20T12:11:38Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1002.eqiad.wmnet 2471401 wikitext text/x-wiki 1pufevkz8204seccfs9szmwypn7g8fs 2471402 2471401 2026-10-20T12:13:47Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2009.codfw.wmnet 2471402 wikitext text/x-wiki 6o5vs7hx3j4g4nv4jkc8ejy468r33om 2471403 2471402 2026-10-20T12:13:51Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2012.codfw.wmnet 2471403 wikitext text/x-wiki 2abmcnro6at3takjsdachohuckdnwml 2471404 2471403 2026-10-20T12:13:56Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96881 and previous config saved to /var/cache/conftool/dbconfig/20261020-121354-ladsgroup.json 2471404 wikitext text/x-wiki jqjq6vmjurp67d1jpzjpiis3mmdols5 2471405 2471404 2026-10-20T12:14:04Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbproxy1023.eqiad.wmnet with reason: Reboot 2471405 wikitext text/x-wiki tpzc3qywqkvrrkeixx4ic85aeony6t4 2471406 2471405 2026-10-20T12:14:17Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudvirtlocal1003.eqiad.wmnet 2471406 wikitext text/x-wiki iojis4kzfoxw7txnptlg64dx9ixnuap 2471407 2471406 2026-10-20T12:16:48Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2012.codfw.wmnet 2471407 wikitext text/x-wiki 3b591kd2rxst3c74mfju7c9q7vl37u9 2471408 2471407 2026-10-20T12:16:52Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup2014.codfw.wmnet 2471408 wikitext text/x-wiki klvccengg3ugfw7z9fm4c0j96nomo7y 2471409 2471408 2026-10-20T12:20:53Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup2014.codfw.wmnet 2471409 wikitext text/x-wiki sn38is8fft1ecamwllsdleak1z40h38 2471410 2471409 2026-10-20T12:21:07Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirtlocal1003.eqiad.wmnet 2471410 wikitext text/x-wiki 4y87zb6fjin3scf8dw8cpkms9gulmwv 2471413 2471410 2026-10-20T12:23:58Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96882 and previous config saved to /var/cache/conftool/dbconfig/20261020-122356-ladsgroup.json 2471413 wikitext text/x-wiki agwzkwjho2rm24663ssejva7v8gc2vq 2471414 2471413 2026-10-20T12:24:02Z Stashbot 7414 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host backup1014.eqiad.wmnet 2471414 wikitext text/x-wiki 8n0zgjvsf6obzlfc8fk1o909lm5jt0q 2471415 2471414 2026-10-20T12:24:45Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host ceph-admin1001.eqiad.wmnet 2471415 wikitext text/x-wiki cat2zsqq81ix8vlz0f8zn94iei4vzv6 2471416 2471415 2026-10-20T12:28:18Z Stashbot 7414 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host backup1014.eqiad.wmnet 2471416 wikitext text/x-wiki beyopgtfqqscnsc2v7qf76aq3nr2bpd 2471417 2471416 2026-10-20T12:28:33Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ceph-admin1001.eqiad.wmnet 2471417 wikitext text/x-wiki 6gdpgosjebnn8x8coyz9kc65hpo3yr4 2471418 2471417 2026-10-20T12:29:41Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad 2471418 wikitext text/x-wiki czszpjr8ufeduiaqmwgprc007kn8tm9 2471419 2471418 2026-10-20T12:32:08Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dbstore1010.eqiad.wmnet 2471419 wikitext text/x-wiki 5m4xf8gsplju2rfdhu9y8mx5quasmy4 2471420 2471419 2026-10-20T12:34:00Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P96883 and previous config saved to /var/cache/conftool/dbconfig/20261020-123357-ladsgroup.json 2471420 wikitext text/x-wiki brtr1z88ausrkt8ou19p1wa5dyw7d1f 2471421 2471420 2026-10-20T12:37:26Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dbstore1010.eqiad.wmnet 2471421 wikitext text/x-wiki s3fof5mimzedonrcvczfszhgjl3j1ph 2471426 2471421 2026-10-20T12:44:02Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T441633)', diff saved to https://phabricator.wikimedia.org/P96884 and previous config saved to /var/cache/conftool/dbconfig/20261020-124359-ladsgroup.json 2471426 wikitext text/x-wiki rlekc7h0pup41cli3c27be1n4nafw1m 2471427 2471426 2026-10-20T12:44:22Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2224.codfw.wmnet with reason: Maintenance 2471427 wikitext text/x-wiki tv3f43w1th7whdsc5bddrizd34490ax 2471428 2471427 2026-10-20T12:44:26Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96885 and previous config saved to /var/cache/conftool/dbconfig/20261020-124421-ladsgroup.json 2471428 wikitext text/x-wiki aj9a4d12b8r48q8y1fe65texew8mj7u 2471430 2471428 2026-10-20T12:51:15Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96886 and previous config saved to /var/cache/conftool/dbconfig/20261020-125112-ladsgroup.json 2471430 wikitext text/x-wiki iealox7juz2qs5fv28634jxwv31swyv 2471431 2471430 2026-10-20T12:55:32Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1001.eqiad.wmnet 2471431 wikitext text/x-wiki 7f4bye8ei33uztfplmy00bln07evm1a 2471432 2471431 2026-10-20T12:58:04Z Stashbot 7414 sukhe@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox) 2471432 wikitext text/x-wiki 4bgvuox0wtd5i9hkx50182ngzsm594o 2471433 2471432 2026-10-20T12:58:08Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org 2471433 wikitext text/x-wiki lghh68v8cpohtk97c9jsvsv3j3wbtvn 2471434 2471433 2026-10-20T13:00:28Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply 2471434 wikitext text/x-wiki qt427lq0c098a0zdopnh1x6krbmd2b8 2471435 2471434 2026-10-20T13:00:59Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply 2471435 wikitext text/x-wiki ku212ooxrbgfyqil0fvugt4sm3zpc3v 2471436 2471435 2026-10-20T13:01:16Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96887 and previous config saved to /var/cache/conftool/dbconfig/20261020-130114-ladsgroup.json 2471436 wikitext text/x-wiki s99mj8wa3zgwfjvwghepg5hs26nsq8y 2471437 2471436 2026-10-20T13:01:20Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1001.eqiad.wmnet 2471437 wikitext text/x-wiki 4ni9vudprddwuvulkrpr7qymus408vp 2471438 2471437 2026-10-20T13:03:34Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1002.eqiad.wmnet 2471438 wikitext text/x-wiki 78kdsnix9vrgd23u1r9hf5dx25m5plo 2471439 2471438 2026-10-20T13:09:14Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1001.eqiad.wmnet 2471439 wikitext text/x-wiki nz1i66msa9kxrn81zukcynxxu7xqmq5 2471440 2471439 2026-10-20T13:09:41Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1002.eqiad.wmnet 2471440 wikitext text/x-wiki pc56frqbn0yyg4m3dlfwurgbonw1k4r 2471441 2471440 2026-10-20T13:10:42Z Stashbot 7414 filippo@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudrabbit1003.eqiad.wmnet 2471441 wikitext text/x-wiki ppvr57i86gvku1ppyf5exctliaq4qzf 2471442 2471441 2026-10-20T13:11:18Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P96888 and previous config saved to /var/cache/conftool/dbconfig/20261020-131116-ladsgroup.json 2471442 wikitext text/x-wiki mzkrch5wbgksv3hpfwk1oxe1bqfkuyc 2471444 2471442 2026-10-20T13:14:30Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1001.eqiad.wmnet 2471444 wikitext text/x-wiki 13evsq5yqr4h1uyfp6qe42f4f61gumz 2471445 2471444 2026-10-20T13:14:38Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org 2471445 wikitext text/x-wiki 72v9a23i1pv5q4n16bk6wx284eue8n0 2471446 2471445 2026-10-20T13:16:50Z Stashbot 7414 filippo@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudrabbit1003.eqiad.wmnet 2471446 wikitext text/x-wiki 6noqpvkoip52kf01wguyxm68r4ru6b5 2471447 2471446 2026-10-20T13:19:28Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1002.eqiad.wmnet 2471447 wikitext text/x-wiki 1du4qndhqpsa2bibjlc4ye00pq8btm3 2471448 2471447 2026-10-20T13:20:06Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet 2471448 wikitext text/x-wiki jv0xxhyupwsqeln864naoxuuwfocf2j 2471449 2471448 2026-10-20T13:21:20Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T441633)', diff saved to https://phabricator.wikimedia.org/P96889 and previous config saved to /var/cache/conftool/dbconfig/20261020-132118-ladsgroup.json 2471449 wikitext text/x-wiki 4d48dgh5fl9jcptf67u0919fvwb7811 2471450 2471449 2026-10-20T13:22:32Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet 2471450 wikitext text/x-wiki m7dspjmfb5p765a5bzpaz3fvbl6o0ra 2471451 2471450 2026-10-20T13:22:53Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet 2471451 wikitext text/x-wiki 1dpc4ksv7dv7tdzhbkwsvw0am388kbi 2471452 2471451 2026-10-20T13:25:04Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1002.eqiad.wmnet 2471452 wikitext text/x-wiki j3ls654zwd9khmfgzvdwruo6223op8v 2471453 2471452 2026-10-20T13:25:13Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1003.eqiad.wmnet 2471453 wikitext text/x-wiki 7j0k5x1svlrwznomjnr2za5p7sq39er 2471454 2471453 2026-10-20T13:25:19Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet 2471454 wikitext text/x-wiki gttvzqgbp2glm014kn5oz7rhgx2i5e6 2471455 2471454 2026-10-20T13:25:28Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet 2471455 wikitext text/x-wiki jp1c81r88sera04jfvxfirflcbrqhu4 2471456 2471455 2026-10-20T13:27:22Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad 2471456 wikitext text/x-wiki lcnh43yvdq16048awpujhtr1rvwisti 2471457 2471456 2026-10-20T13:27:52Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet 2471457 wikitext text/x-wiki 9dbyy98ki8qbm715uxnl1xb0netvb6o 2471458 2471457 2026-10-20T13:27:58Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet 2471458 wikitext text/x-wiki 2ikfe3ff4ktd82g0enx4xtnyv94ceis 2471459 2471458 2026-10-20T13:29:37Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org 2471459 wikitext text/x-wiki ithqwoed4ntcttddolm4qt9twnr0exz 2471460 2471459 2026-10-20T13:29:58Z Stashbot 7414 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 18 hosts with reason: Restart 2471460 wikitext text/x-wiki ga0m360bsfxzhmkm0unomlrutuf245w 2471461 2471460 2026-10-20T13:30:35Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1003.eqiad.wmnet 2471461 wikitext text/x-wiki swwncu9w4kb4qk5i7rn5ys9dkntkpe9 2471462 2471461 2026-10-20T13:31:22Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet 2471462 wikitext text/x-wiki hnv9se4ewcd6nnr9uw4myh688l39pr0 2471463 2471462 2026-10-20T13:31:55Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet 2471463 wikitext text/x-wiki sqirmjvbe5q4b5nq8npqm5lefmm4s9k 2471464 2471463 2026-10-20T13:35:18Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet 2471464 wikitext text/x-wiki rm889fgvcyvqo599r71dq25e3w7jz9v 2471465 2471464 2026-10-20T13:35:36Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet 2471465 wikitext text/x-wiki 7valhi0u2jekhddwfytwya5t76duj3a 2471466 2471465 2026-10-20T13:37:11Z Stashbot 7414 vgutierrez: upload liberica 0.28 to trixie-wikimedia (apt.wm.o) - T401832 2471466 wikitext text/x-wiki d1gt50tfr8zpywvtjou669nno38lb2i 2471467 2471466 2026-10-20T13:38:53Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'configure' for AS: 44034 2471467 wikitext text/x-wiki 9od0p1p5ip3d824uin8bpqj5wqo7sgy 2471468 2471467 2026-10-20T13:39:28Z Stashbot 7414 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'configure' for AS: 44034 2471468 wikitext text/x-wiki fs8c83vgrs3s4ik39yj7qwytutc073i 2471469 2471468 2026-10-20T13:39:35Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet 2471469 wikitext text/x-wiki 6omz6oghiq7i26vo3bbpm749kbx0c5q 2471470 2471469 2026-10-20T13:40:41Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.reimage for host lvs7003.magru.wmnet with OS trixie 2471470 wikitext text/x-wiki scmr0u3taj7nhopj5y824427u72uaj0 2471471 2471470 2026-10-20T13:41:28Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet 2471471 wikitext text/x-wiki p07qm47oxkulx0ll105h6jphin9uwm4 2471473 2471471 2026-10-20T13:43:52Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet 2471473 wikitext text/x-wiki amzbyizlxzupsw74quhrj5pu0pevjkg 2471474 2471473 2026-10-20T13:43:59Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet 2471474 wikitext text/x-wiki 6s7yhpml85bskaeogr45wiekumczz5f 2471475 2471474 2026-10-20T13:45:37Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org 2471475 wikitext text/x-wiki n0w6nks2keys6oiocsizcapuxopcz8b 2471477 2471475 2026-10-20T13:46:27Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet 2471477 wikitext text/x-wiki hdxo7qf1grdf656r3kre6u3ybxnp06y 2471480 2471477 2026-10-20T13:46:52Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad 2471480 wikitext text/x-wiki 5d9uhjb7hkpqnw3uvowf7dgafgohahw 2471481 2471480 2026-10-20T13:47:03Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet 2471481 wikitext text/x-wiki otvmwp1pbfx84r2nfolniz1ekuooffs 2471482 2471481 2026-10-20T13:49:25Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet 2471482 wikitext text/x-wiki nlqmem0yquye81n7woyh76lzy9h7s9e 2471483 2471482 2026-10-20T13:49:34Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet 2471483 wikitext text/x-wiki 0wtst6bd7zivszibfoq7n5nmbscfj1s 2471484 2471483 2026-10-20T13:51:44Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet 2471484 wikitext text/x-wiki 4fszmgt9tx4mi93ixxy19x1afi80jj6 2471485 2471484 2026-10-20T13:51:51Z Stashbot 7414 dkertesz@cumin1004: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435) 2471485 wikitext text/x-wiki lcld6ujjijffi7rrn21xt3pmw5bevja 2471487 2471485 2026-10-20T13:52:35Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet 2471487 wikitext text/x-wiki j063ppikler354xt9fdcwpzdk0dsyw0 2471489 2471487 2026-10-20T13:54:40Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw 2471489 wikitext text/x-wiki 9qjeoiyixpzc56v05st9kp664v83hrg 2471490 2471489 2026-10-20T13:54:58Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet 2471490 wikitext text/x-wiki jqic0lbwjhp8aese4geeq5j6jesi7dn 2471491 2471490 2026-10-20T13:55:17Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1001.eqiad.wmnet 2471491 wikitext text/x-wiki tgh71qx93jeppo2kwy2plalx0w9yemh 2471492 2471491 2026-10-20T13:58:13Z Stashbot 7414 a-pizzata@deploy1003: Started deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4] 2471492 wikitext text/x-wiki bdfbtexy1nn1eba5euu81pqb9svx89n 2471493 2471492 2026-10-20T13:58:39Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1001.eqiad.wmnet 2471493 wikitext text/x-wiki 3t6h0h6u39fiai0h3dvai1n5m6lf836 2471494 2471493 2026-10-20T13:58:46Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl1002.eqiad.wmnet 2471494 wikitext text/x-wiki 0yccz4uxncdens8bvokgrxxkdzd12e0 2471495 2471494 2026-10-20T13:59:00Z Stashbot 7414 a-pizzata@deploy1003: Finished deploy [analytics/refinery@68687dc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@68687dc4] (duration: 00m 46s) 2471495 wikitext text/x-wiki czr8gm24s4kooyfhgf49b6jcamichyd 2471496 2471495 2026-10-20T13:59:15Z Stashbot 7414 a-pizzata@deploy1003: Started deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4] 2471496 wikitext text/x-wiki dtisdmt2e9xuf3ywqxe4u0poej15us6 2471497 2471496 2026-10-20T14:00:37Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org 2471497 wikitext text/x-wiki h76mc7ittbvat00yxzua7oxko3a85sd 2471499 2471497 2026-10-20T14:02:08Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl1002.eqiad.wmnet 2471499 wikitext text/x-wiki f4s1yh7jqgt2pohp5gh4jyl1285mtd3 2471500 2471499 2026-10-20T14:03:09Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet 2471500 wikitext text/x-wiki bbsgpk8d45yvaqxh78y3uayv3lofh3v 2471501 2471500 2026-10-20T14:03:42Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage 2471501 wikitext text/x-wiki 8ivqbnk3p52sb3mxbzytgh1tz1wsahk 2471502 2471501 2026-10-20T14:04:55Z Stashbot 7414 a-pizzata@deploy1003: Finished deploy [analytics/refinery@68687dc]: Regular analytics weekly train [analytics/refinery@68687dc4] (duration: 05m 39s) 2471502 wikitext text/x-wiki a9mdllkmxng5fl1agk50fxymtt2m2aq 2471504 2471502 2026-10-20T14:06:55Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2471504 wikitext text/x-wiki 1vttpxk36fhikk8fuwif8tcnle11bd0 2471506 2471504 2026-10-20T14:07:16Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471506 wikitext text/x-wiki g9gcjuo00fw0d3q76fzv3c0tkb61272 2471507 2471506 2026-10-20T14:08:05Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2001.codfw.wmnet 2471507 wikitext text/x-wiki 1p7whtp47ph60givclvpu15qi3ey4i2 2471508 2471507 2026-10-20T14:08:15Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet 2471508 wikitext text/x-wiki od7hulha8z4e9e4a64cauhi3xkwghcl 2471510 2471508 2026-10-20T14:08:27Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs7003.magru.wmnet with reason: host reimage 2471510 wikitext text/x-wiki 4k0vkii9qmke1xdi92clmlpegnu4atv 2471511 2471510 2026-10-20T14:10:07Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2471511 wikitext text/x-wiki cgf4dmgh99cwwgzyxvm3yoi0eymcsji 2471513 2471511 2026-10-20T14:10:25Z Stashbot 7414 sukhe@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org 2471513 wikitext text/x-wiki hxq4fctgp9ikhbm10ubjrkwvz91wd5b 2471514 2471513 2026-10-20T14:10:29Z Stashbot 7414 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on P{dns7002* or dns6001* or dns4003*} and (A:dnsbox) 2471514 wikitext text/x-wiki mjozj2jj4912h9ky6hd8hgovaphywdt 2471515 2471514 2026-10-20T14:10:47Z Stashbot 7414 hashar@deploy1003: Started deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088 2471515 wikitext text/x-wiki gfj8oygpm1bwhvby6u2kqg778vwogyo 2471516 2471515 2026-10-20T14:10:58Z Stashbot 7414 hashar@deploy1003: Finished deploy [integration/docroot@5cf09c8]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s) 2471516 wikitext text/x-wiki 32i4kx7tbdjgn0gs76jxvmq8e349u7v 2471517 2471516 2026-10-20T14:11:55Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw 2471517 wikitext text/x-wiki 2r4xoeihfn4x9qf0lmtbwbyellhsq68 2471518 2471517 2026-10-20T14:12:42Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad 2471518 wikitext text/x-wiki aw9kg1kpecju03ow2adaqzeqka7usjc 2471519 2471518 2026-10-20T14:13:12Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet 2471519 wikitext text/x-wiki 5v9jlq4y67orh64eyvpzu27jzdbdgro 2471521 2471519 2026-10-20T14:13:51Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471521 wikitext text/x-wiki lf9cf5e9egcx1s8hioxvt7a7bsxy6wa 2471523 2471521 2026-10-20T14:15:25Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:ml-staging-worker 2471523 wikitext text/x-wiki 8xy77dayy5u7bk0if0ujzkrv1dl9j1f 2471524 2471523 2026-10-20T14:15:30Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2001.codfw.wmnet 2471524 wikitext text/x-wiki qdd9zkqo6jsutctngmov5mi8mpz778d 2471526 2471524 2026-10-20T14:17:27Z Stashbot 7414 hashar@deploy1003: Started deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088 2471526 wikitext text/x-wiki 4e34j3ontandb9obnzj5qhh17ookltu 2471527 2471526 2026-10-20T14:17:37Z Stashbot 7414 hashar@deploy1003: Finished deploy [integration/docroot@be38d5a]: opensource: docker-pkg has moved to GitLab - T441088 (duration: 00m 10s) 2471527 wikitext text/x-wiki cv4mkbtvo7z2uuf6gxx9b45179xgznk 2471530 2471527 2026-10-20T14:18:46Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471530 wikitext text/x-wiki pmin7phuwb3v3dwi6voevkbky4xcokf 2471531 2471530 2026-10-20T14:18:50Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471531 wikitext text/x-wiki onhmc86bh3kh37dqy7sjlytk9ppe4ls 2471532 2471531 2026-10-20T14:19:00Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471532 wikitext text/x-wiki skxmbh9frqqyylduqnzpwdupopj8mn1 2471533 2471532 2026-10-20T14:19:04Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471533 wikitext text/x-wiki 4m1a3r9cccwsmgcl29cjecbirtduqh5 2471534 2471533 2026-10-20T14:19:10Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471534 wikitext text/x-wiki mo0osiut3y1yq8aob4xjbxi0pnoypie 2471535 2471534 2026-10-20T14:19:13Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2471535 wikitext text/x-wiki obih94kjmtnq1adry1smawvm4n9rwjw 2471537 2471535 2026-10-20T14:20:21Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471537 wikitext text/x-wiki 40bpqfh4je11mocyj7qk3bbwb7qte1x 2471538 2471537 2026-10-20T14:20:29Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' . 2471538 wikitext text/x-wiki 6apq8q3u8v9m1ubyo676wig4fr4kiwz 2471539 2471538 2026-10-20T14:20:33Z Stashbot 7414 bwojtowicz@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' . 2471539 wikitext text/x-wiki m37y216nsbqwi5e1587d2kj79rmnprt 2471542 2471539 2026-10-20T14:22:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2471542 wikitext text/x-wiki m4lqhvsoqp2v3mz52bqxnqze5odl896 2471544 2471542 2026-10-20T14:24:52Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw 2471544 wikitext text/x-wiki d8oqez4xcl3gk006ie3wd5jv65lt59k 2471545 2471544 2026-10-20T14:25:33Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2001.codfw.wmnet 2471545 wikitext text/x-wiki 7emm47ol1o5k7mpbx12h2kvpp4bojrf 2471546 2471545 2026-10-20T14:25:39Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471546 wikitext text/x-wiki k8d4y2x0nqjlyo7q0ar427gr27fdnpl 2471548 2471546 2026-10-20T14:26:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2471548 wikitext text/x-wiki oqmuqmzkm5olx82oh1yf6tc3irc1ojw 2471550 2471548 2026-10-20T14:29:14Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471550 wikitext text/x-wiki p4in1m32fz88v57i6kghbw4x2ormypv 2471551 2471550 2026-10-20T14:29:23Z Stashbot 7414 ayounsi@cumin1004: START - Cookbook sre.network.sync-bgp-netbox 2471551 wikitext text/x-wiki 0026n2xdl9c3ntktfc34pwmfp74f6l8 2471552 2471551 2026-10-20T14:29:27Z Stashbot 7414 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.sync-bgp-netbox (exit_code=0) 2471552 wikitext text/x-wiki oar9j4cq1rp5f3h9kqyqpinojb9ruvr 2471553 2471552 2026-10-20T14:29:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471553 wikitext text/x-wiki nf2a7kq4zt65hayua8ww9l9bf8alt1c 2471554 2471553 2026-10-20T14:29:39Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471554 wikitext text/x-wiki 80j0rczlbqlw9g1s1n77icj6xq5cc96 2471555 2471554 2026-10-20T14:29:45Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply 2471555 wikitext text/x-wiki jomcq1lyjxgk5bsz0qgvkqlq75bahix 2471556 2471555 2026-10-20T14:30:01Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply 2471556 wikitext text/x-wiki 80dc62gykgh3wrei1jv0s94yw0tn71h 2471557 2471556 2026-10-20T14:30:21Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2001.codfw.wmnet 2471557 wikitext text/x-wiki cafd1eo96dud1y1juv2842qviclc0dk 2471558 2471557 2026-10-20T14:30:25Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2001.codfw.wmnet 2471558 wikitext text/x-wiki ltotrh0ot7s66v6zcaeogxyljyd1ylb 2471559 2471558 2026-10-20T14:30:29Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2002.codfw.wmnet 2471559 wikitext text/x-wiki 9gc5a2uw32yzy5dan1zc191xo1f51o7 2471560 2471559 2026-10-20T14:32:07Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad 2471560 wikitext text/x-wiki dttwzclzii85e3hqo4coo39t1z7he0c 2471561 2471560 2026-10-20T14:34:13Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] START helmfile.d/services/mw-web: apply 2471561 wikitext text/x-wiki iv81u9xggoypjsbwbf6zcxcfdnt0wsc 2471562 2471561 2026-10-20T14:34:32Z Stashbot 7414 jiji@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-web: apply 2471562 wikitext text/x-wiki tr6rd134mi9qtb9cu6gopqza3fqa0m7 2471563 2471562 2026-10-20T14:37:13Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471563 wikitext text/x-wiki lxw42cserr24lh1hl3yctwsdo9atf3j 2471564 2471563 2026-10-20T14:37:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2471564 wikitext text/x-wiki 9rc7t5lnx55xrole3kvdtxk6mcmryaa 2471565 2471564 2026-10-20T14:37:54Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply 2471565 wikitext text/x-wiki 2bfcvlcpfwn9jex2u7eq2p6g7b3qzrh 2471566 2471565 2026-10-20T14:37:58Z Stashbot 7414 cscott@deploy1003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply 2471566 wikitext text/x-wiki gvih850m3wf9k3onieobcvezk2sbd5e 2471567 2471566 2026-10-20T14:38:18Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471567 wikitext text/x-wiki fwndqd1cj0geyl2r8o3pvwpdmxw224y 2471568 2471567 2026-10-20T14:39:18Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471568 wikitext text/x-wiki 8pzreqrsr5r8wv2rugb3f0gbf27kgae 2471569 2471568 2026-10-20T14:40:34Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2002.codfw.wmnet 2471569 wikitext text/x-wiki mycqhdujrpld2f4oinw7yi7mxthf6ss 2471570 2471569 2026-10-20T14:40:46Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs7003.magru.wmnet with OS trixie 2471570 wikitext text/x-wiki m9ejm3j38tbsyzgr6cblbi7d26sv1cv 2471571 2471570 2026-10-20T14:41:33Z Stashbot 7414 vgutierrez: upload ipip-multiqueue-optimizer 0.3+deb13u1 to trixie-wikimedia (apt.wm.o) - T401832 2471571 wikitext text/x-wiki romloue653w6qs4v5igha5ko9n82isn 2471572 2471571 2026-10-20T14:43:53Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471572 wikitext text/x-wiki 3npcz01rkzu94m25oxa5wwv07pvbpht 2471573 2471572 2026-10-20T14:44:18Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw 2471573 wikitext text/x-wiki r5prhgz50xfmdzb504apz7c59pbgbjg 2471574 2471573 2026-10-20T14:45:23Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2002.codfw.wmnet 2471574 wikitext text/x-wiki sml2cyl0hfwh220cwwjrasr6snst0b8 2471575 2471574 2026-10-20T14:45:28Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2002.codfw.wmnet 2471575 wikitext text/x-wiki jhcjcj1243rjhqlkwznbas9drwkv8xo 2471576 2471575 2026-10-20T14:45:32Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-staging2003.codfw.wmnet 2471576 wikitext text/x-wiki 5zviyop212yib0wiysddaqz6hcxion9 2471577 2471576 2026-10-20T14:45:49Z Stashbot 7414 elukey@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad 2471577 wikitext text/x-wiki gzkmi4rjwiemn4wivjaqpume6t6t2hu 2471578 2471577 2026-10-20T14:47:31Z Stashbot 7414 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: T440896 2471578 wikitext text/x-wiki rycnr6hku3a786llclt4z6a5tg5s94k 2471579 2471578 2026-10-20T14:48:05Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471579 wikitext text/x-wiki f8122aoyjej0ue9tuc1jy8e8mqpw6mh 2471580 2471579 2026-10-20T14:52:14Z Stashbot 7414 dkertesz@cumin1004: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp7001.*} and A:cp - upgrade haproxy to 3.2.26-1 (T441435) 2471580 wikitext text/x-wiki 6u886ud8g81lwtuhz36y2krv930thc1 2471581 2471580 2026-10-20T14:52:36Z Stashbot 7414 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool mw-web-ro in eqiad: T440896 2471581 wikitext text/x-wiki jkoo5akz7oo0lnedyolebc1gjmz0jr6 2471582 2471581 2026-10-20T14:55:38Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-staging2003.codfw.wmnet 2471582 wikitext text/x-wiki h9m20enquwvm9vjbbf8m394ut8a3jpp 2471583 2471582 2026-10-20T14:57:20Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin depooling P{lvs7002.*} and A:liberica 2471583 wikitext text/x-wiki bcwc80emcprc91phvw781i2a51q1nok 2471584 2471583 2026-10-20T14:57:28Z Stashbot 7414 effie: depooled eqiad for mw-web T440896 2471584 wikitext text/x-wiki on6vz3t45x7je8c9grdehb3ijxrzxiv 2471585 2471584 2026-10-20T14:57:32Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling P{lvs7002.*} and A:liberica 2471585 wikitext text/x-wiki hl89bkzq888xbws7juc8becp5nrwavn 2471586 2471585 2026-10-20T14:57:48Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply 2471586 wikitext text/x-wiki 18mbx9abanz1ypzw065tezaye99l0hp 2471587 2471586 2026-10-20T14:57:54Z Stashbot 7414 cscott@deploy1003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply 2471587 wikitext text/x-wiki ix4nrzfqhy3ydxiaekb2apx9s1is35k 2471588 2471587 2026-10-20T14:58:45Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471588 wikitext text/x-wiki ggteso9wl91mvya0az33vlz0415inzx 2471589 2471588 2026-10-20T15:00:06Z Stashbot 7414 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-staging2003.codfw.wmnet 2471589 wikitext text/x-wiki lj7kuzdjwjldpyn86l6il3h9qyvtfgq 2471590 2471589 2026-10-20T15:00:10Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-staging2003.codfw.wmnet 2471590 wikitext text/x-wiki d8sbax01ygqnvtp6kvlcad38jqvvjyk 2471591 2471590 2026-10-20T15:00:14Z Stashbot 7414 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:ml-staging-worker 2471591 wikitext text/x-wiki gt7m215x3fthyk2o4y2bd9yelix8aby 2471592 2471591 2026-10-20T15:04:34Z Stashbot 7414 dancy@deploy1003: Started scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222 2471592 wikitext text/x-wiki a6d4atw92bxh4j3izi565w0dp800owz 2471593 2471592 2026-10-20T15:05:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471593 wikitext text/x-wiki olagsrz82njrnb3szgqvge0s4yy9b7j 2471594 2471593 2026-10-20T15:05:27Z Stashbot 7414 elukey@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad 2471594 wikitext text/x-wiki pht3ekof1b301683g3tmgie7pxvpalt 2471595 2471594 2026-10-20T15:13:16Z Stashbot 7414 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin pooling P{lvs7002.*} and A:liberica 2471595 wikitext text/x-wiki rqirl0qapq1rb42srasz2cejskgp8rs 2471596 2471595 2026-10-20T15:13:41Z Stashbot 7414 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) pooling P{lvs7002.*} and A:liberica 2471596 wikitext text/x-wiki 9ye3cxpysv690dsmuv9t08zrcsgct5e 2471597 2471596 2026-10-20T15:21:45Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.provision for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2471597 wikitext text/x-wiki f7uhoa85q8max3q0t3h28dndjxwurze 2471598 2471597 2026-10-20T15:23:33Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471598 wikitext text/x-wiki gws2tz4x6movpicmo49ei56wkhcf3z0 2471599 2471598 2026-10-20T15:23:46Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471599 wikitext text/x-wiki stlf57baiwypqc4q465qqfd987dvqk1 2471600 2471599 2026-10-20T15:27:18Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host druid-internal1004.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART 2471600 wikitext text/x-wiki 3fe0trscksc9ti09higxoduzm7k10o7 2471601 2471600 2026-10-20T15:31:03Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2471601 wikitext text/x-wiki ltr5uxapsu4l6ua16243it84y41u34w 2471602 2471601 2026-10-20T15:31:41Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2471602 wikitext text/x-wiki 35os6ojuhivn9ypgpqm791kd6kfb660 2471603 2471602 2026-10-20T15:32:01Z Stashbot 7414 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe 2471603 wikitext text/x-wiki 7ck9ima9m66nyknx96v7c0a59bzpnj8 2471604 2471603 2026-10-20T15:32:22Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply 2471604 wikitext text/x-wiki jwl7lhd55amyvihqvzrbom7wbueumem 2471605 2471604 2026-10-20T15:32:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe 2471605 wikitext text/x-wiki n8fwjm91gwak5gm7mp56v195ut5yeee 2471606 2471605 2026-10-20T15:36:42Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe 2471606 wikitext text/x-wiki pfx0hvk88ltz5exqrp0k9cev8kzfb7y 2471607 2471606 2026-10-20T15:41:42Z Stashbot 7414 dancy@deploy1003: Finished scap sync-world: testwikis to 1.48.0-wmf.1 refs T438222 (duration: 37m 07s) 2471607 wikitext text/x-wiki 0p9m6m8b978dc110xzngiienuf2axhx 2471608 2471607 2026-10-20T15:42:03Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply 2471608 wikitext text/x-wiki 4ete80i44nd1z24hjyy943csn1cg5gc 2471609 2471608 2026-10-20T15:43:02Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply 2471609 wikitext text/x-wiki epxbyvt68hj5bsxxh7lz7jvuekfa2me 2471611 2471609 2026-10-20T15:44:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply 2471611 wikitext text/x-wiki ssajsj52skdc8zkjmi6vmpbnis0bxhc 2471613 2471611 2026-10-20T15:46:59Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2471613 wikitext text/x-wiki tkaokc16bg3yb30w0j04lncog7qb9l4 2471614 2471613 2026-10-20T15:47:30Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2471614 wikitext text/x-wiki 8r4nsdskzmloag3y516bvha1yt8tf6q 2471615 2471614 2026-10-20T15:48:04Z Stashbot 7414 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe 2471615 wikitext text/x-wiki lippulq6e5zujvxpw9acnuqn87fk9yd 2471616 2471615 2026-10-20T15:48:10Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply 2471616 wikitext text/x-wiki oathm594k7s4k7wq6j0v2moq106u1lx 2471617 2471616 2026-10-20T15:48:33Z Stashbot 7414 ryankemper: T435477 `[airflow-analytics-test]` gearing up to start upgrade to Airflow 3.3, pausing all associated DAGs shortly 2471617 wikitext text/x-wiki gpfu443hcbl78546r6983m531ep7omo 2471618 2471617 2026-10-20T15:48:43Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply 2471618 wikitext text/x-wiki 759nb6h0dib8eyikwtfzkcqtbcmm177 2471619 2471618 2026-10-20T15:51:20Z Stashbot 7414 marostegui@dns1004: START - running authdns-update 2471619 wikitext text/x-wiki f53ex5xlxyqa17lxvktfbrddfg0g88n 2471620 2471619 2026-10-20T15:52:04Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2471620 wikitext text/x-wiki b7a99qlbqrfmlo4e4o6b31p22j5s6zp 2471621 2471620 2026-10-20T15:53:45Z Stashbot 7414 marostegui@dns1004: END - running authdns-update 2471621 wikitext text/x-wiki btbuc5bfq9rhcdm0s83gbfzgfp1pymh 2471622 2471621 2026-10-20T15:54:25Z Stashbot 7414 ryankemper: T435477 `[airflow-analytics-test]` all associated DAGs paused 2471622 wikitext text/x-wiki j7mdl9d0lbh7k92u0xwm90c96aaia27 2471623 2471622 2026-10-20T15:55:18Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2471623 wikitext text/x-wiki h70jkbrdv66s02wj1h4cviy3phomts3 2471624 2471623 2026-10-20T15:56:00Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply 2471624 wikitext text/x-wiki fvk88rwz988k2c5nm8hxlkw0srzybu2 2471625 2471624 2026-10-20T15:59:21Z Stashbot 7414 herron@cumin1004: START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:logstash-eqiad 2471625 wikitext text/x-wiki 1cpj9vwkd1eao61yzojfyfzeo55g75h 2471626 2471625 2026-10-20T15:59:25Z Stashbot 7414 trueg@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply 2471626 wikitext text/x-wiki 3qb3kueru4sixb7olyrmklzb9u71imo 2471627 2471626 2026-10-20T16:06:07Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1165.eqiad.wmnet with reason: Maintenance 2471627 wikitext text/x-wiki 2mskzw9ljjdn42iu5i7711ldw6gumi7 2471628 2471627 2026-10-20T16:06:12Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96892 and previous config saved to /var/cache/conftool/dbconfig/20261020-160606-ladsgroup.json 2471628 wikitext text/x-wiki c6dtg86tpbto3zcdsznbwf4kem0f07s 2471630 2471628 2026-10-20T16:09:08Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2471630 wikitext text/x-wiki 87qgtzl7f6p93uzvfva5vubjwh5nhyp 2471631 2471630 2026-10-20T16:10:03Z Stashbot 7414 ryankemper@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply 2471631 wikitext text/x-wiki mu1c4o1g5lwfpkkiruq7hfgeipysb7j 2471632 2471631 2026-10-20T16:11:02Z Stashbot 7414 ryankemper: T435477 `[airflow-analytics-test]` applying https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1357219, so this marks the official cutover 2471632 wikitext text/x-wiki 1mg3t4ss0alww2mexp7vz0t8ixlmnq3 2471633 2471632 2026-10-20T16:12:07Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471633 wikitext text/x-wiki qng5hy7r3n3amw43mct9dpcrsmyv9vc 2471634 2471633 2026-10-20T16:13:20Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96893 and previous config saved to /var/cache/conftool/dbconfig/20261020-161317-ladsgroup.json 2471634 wikitext text/x-wiki 17zwr092kotsijjwt6dyqcoij0x5ubo 2471637 2471634 2026-10-20T16:23:21Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96895 and previous config saved to /var/cache/conftool/dbconfig/20261020-162319-ladsgroup.json 2471637 wikitext text/x-wiki s1z8d0fl5c6qq5pysjwwxvat3uogzg8 2471639 2471637 2026-10-20T16:25:58Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1005.eqiad.wmnet 2471639 wikitext text/x-wiki n3lbca7xat7ybf0eekwqhvkty6x3ukg 2471640 2471639 2026-10-20T16:27:51Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471640 wikitext text/x-wiki a6wcyt9tkfsf61u8ke2d441iai5ckjn 2471641 2471640 2026-10-20T16:28:03Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471641 wikitext text/x-wiki tcwf6g5ev9cozy6g0l9lw2zqkcrn9rn 2471642 2471641 2026-10-20T16:28:08Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471642 wikitext text/x-wiki gqt4kx64eytbeokmko34xghijam2nkx 2471643 2471642 2026-10-20T16:29:20Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471643 wikitext text/x-wiki 7k57w451b9umdu1xmjj3o1oackddk66 2471644 2471643 2026-10-20T16:31:15Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1005.eqiad.wmnet 2471644 wikitext text/x-wiki k09d8ge6yci9w43wcha77aicjim1l5x 2471645 2471644 2026-10-20T16:32:48Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.dns.netbox 2471645 wikitext text/x-wiki cfr53ihjzdjc4eb82hcf9n43w8tsigy 2471646 2471645 2026-10-20T16:33:23Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P96897 and previous config saved to /var/cache/conftool/dbconfig/20261020-163321-ladsgroup.json 2471646 wikitext text/x-wiki oximotgr5witfa7zsclz8nxsv1808n1 2471647 2471646 2026-10-20T16:33:38Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp 2471647 wikitext text/x-wiki 4vpm4yi461spily3awjpakd13v7qc7g 2471648 2471647 2026-10-20T16:36:32Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host druid-internal1006.eqiad.wmnet 2471648 wikitext text/x-wiki bi6rylce6g4hxzqm6b3aqzge6i3risb 2471649 2471648 2026-10-20T16:37:17Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004" 2471649 wikitext text/x-wiki 9mhk1fkpuopaskcubydsnru7yd8sftk 2471650 2471649 2026-10-20T16:37:21Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries new cloudsw - cmooney@cumin1004" 2471650 wikitext text/x-wiki mjledi7b8znvg44v823yj5tb7yyi1p2 2471651 2471650 2026-10-20T16:37:25Z Stashbot 7414 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) 2471651 wikitext text/x-wiki j5ggonoh8h5tyy6hzieu07f6zyjbmt4 2471652 2471651 2026-10-20T16:37:41Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster 2471652 wikitext text/x-wiki eua6whhxe4t0ghsmd7z657srddtmd2d 2471653 2471652 2026-10-20T16:38:31Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply 2471653 wikitext text/x-wiki d6zw45mwe7loz301uxs8mdngiyrfpma 2471654 2471653 2026-10-20T16:38:35Z Stashbot 7414 cgoubert@deploy1003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply 2471654 wikitext text/x-wiki nnzn16ghlqagwh3zk44p4dh9640o66u 2471655 2471654 2026-10-20T16:41:22Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad 2471655 wikitext text/x-wiki 8bdl6t0x6d9ssf2nlwq5a8qrte8wz4f 2471656 2471655 2026-10-20T16:41:27Z Stashbot 7414 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad 2471656 wikitext text/x-wiki ehrv91tbmuelmk06gjn4ddfsr67nzrq 2471657 2471656 2026-10-20T16:41:47Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host druid-internal1006.eqiad.wmnet 2471657 wikitext text/x-wiki klep4yyiddkwi3rkg07nul9g3eib2yf 2471658 2471657 2026-10-20T16:42:12Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad 2471658 wikitext text/x-wiki mrkjj2pn2z44wirwjnvjr381e8kgwsx 2471659 2471658 2026-10-20T16:42:17Z Stashbot 7414 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad 2471659 wikitext text/x-wiki e6aicabexv1tqxgxqokova13hj3p4r7 2471660 2471659 2026-10-20T16:42:38Z Stashbot 7414 cmooney@cumin1004: START - Cookbook sre.network.tls for network device cloudsw3-c8-eqiad 2471660 wikitext text/x-wiki rdo8exzjr9bhu29wxmft3z446ox53eq 2471661 2471660 2026-10-20T16:42:42Z Stashbot 7414 cmooney@cumin1004: END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device cloudsw3-c8-eqiad 2471661 wikitext text/x-wiki lcg0pak2ab5f218wgc1pbquy7k33sya 2471662 2471661 2026-10-20T16:43:25Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T441633)', diff saved to https://phabricator.wikimedia.org/P96898 and previous config saved to /var/cache/conftool/dbconfig/20261020-164322-ladsgroup.json 2471662 wikitext text/x-wiki bnj2knfdtsbu5n2o23nrswx78o5u68g 2471663 2471662 2026-10-20T16:43:44Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance 2471663 wikitext text/x-wiki 8qdzee9ec9j62kiopviyc0lqyafsjlr 2471664 2471663 2026-10-20T16:43:49Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96899 and previous config saved to /var/cache/conftool/dbconfig/20261020-164343-ladsgroup.json 2471664 wikitext text/x-wiki 25ncwmqszmcqvdkcw2uxjuxpnql3smo 2471665 2471664 2026-10-20T16:45:02Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet 2471665 wikitext text/x-wiki 78s6ffvq43jf0edyax0hy71y75ealps 2471666 2471665 2026-10-20T16:50:47Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96900 and previous config saved to /var/cache/conftool/dbconfig/20261020-165043-ladsgroup.json 2471666 wikitext text/x-wiki 5xv25kqkswjansmam6elwd5rwiq3o06 2471667 2471666 2026-10-20T16:57:34Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad 2471667 wikitext text/x-wiki ch34iwme8up7wlxptd27965st6frcmg 2471668 2471667 2026-10-20T16:59:33Z Stashbot 7414 jelto@deploy1003: Started scap deploy-service miscweb: demo deployment for MVP walk through 2471668 wikitext text/x-wiki 8b7y6d13hxfk5ek01uun17hcy6owqbv 2471669 2471668 2026-10-20T16:59:38Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [STAGING/DEFAULT] staging-eqiad Deploying miscweb 2471669 wikitext text/x-wiki 3xg0rb14lbugv6omwheloeizlo397nj 2471670 2471669 2026-10-20T17:00:36Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] codfw Deploying miscweb 2471670 wikitext text/x-wiki 3a0yjrxsptaoke96yh5ok4kaknysrl1 2471671 2471670 2026-10-20T17:00:48Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96901 and previous config saved to /var/cache/conftool/dbconfig/20261020-170046-ladsgroup.json 2471671 wikitext text/x-wiki 8yrk2zejwgul76mn4yqdtwquwevlpbl 2471672 2471671 2026-10-20T17:01:33Z Stashbot 7414 jelto@deploy1003: scap deploy-service miscweb: [PRODUCTION/DEFAULT] eqiad Deploying miscweb 2471672 wikitext text/x-wiki pzeaka156khutxvuwf3edge2rrl91sv 2471673 2471672 2026-10-20T17:02:11Z Stashbot 7414 jelto@deploy1003: Finished scap deploy-service miscweb: demo deployment for MVP walk through (duration: 02m 37s) 2471673 wikitext text/x-wiki jy8p10aely9dcb7joi4eubq7h5ry5cx 2471674 2471673 2026-10-20T17:02:37Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471674 wikitext text/x-wiki j7wpoh9n2p1gtnp1tot1crqrf2rrg5h 2471675 2471674 2026-10-20T17:03:00Z Stashbot 7414 ryankemper: T435477 `[airflow-analytics-test]` upgrade complete; all previously-unpaused DAGs have been unpaused again. ofc, we'll still watch for anything weird that breaks over the next couple days 2471675 wikitext text/x-wiki q5bxx99d25ngzd0j88esuouuo18n0i4 2471676 2471675 2026-10-20T17:05:04Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471676 wikitext text/x-wiki 6jjvgt2b7t9hqcesnkfayz97s4y1a0o 2471677 2471676 2026-10-20T17:06:27Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad 2471677 wikitext text/x-wiki 4zb6eojt6ggvlebqixldqflv8ko8pyi 2471678 2471677 2026-10-20T17:10:50Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P96903 and previous config saved to /var/cache/conftool/dbconfig/20261020-171048-ladsgroup.json 2471678 wikitext text/x-wiki fn1gijciggi7bb7bbr6b0knsoci7u1x 2471679 2471678 2026-10-20T17:13:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471679 wikitext text/x-wiki 160gowo7800th9sou26d3ryijyuowy1 2471680 2471679 2026-10-20T17:13:20Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471680 wikitext text/x-wiki nkngohdcaeho580p5kxvnhfmnwlsob7 2471681 2471680 2026-10-20T17:13:31Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471681 wikitext text/x-wiki oq8yxcs4zyyelf6ac5h7f6oj2quhyl4 2471682 2471681 2026-10-20T17:13:49Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster 2471682 wikitext text/x-wiki nf6fde6al79pp2ynvb9e5vf7yjj51x8 2471683 2471682 2026-10-20T17:13:53Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster 2471683 wikitext text/x-wiki ce8ss1p7pcij6fojpl3q9uxi8tfebos 2471684 2471683 2026-10-20T17:14:16Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster 2471684 wikitext text/x-wiki 18wlo6hb3wpnvd5o3hqktd5bwwv1sjo 2471685 2471684 2026-10-20T17:14:20Z Stashbot 7414 btullis@cumin1004: END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster 2471685 wikitext text/x-wiki o968pzptwok9c880rrsx6o42hmcvx80 2471686 2471685 2026-10-20T17:15:14Z Stashbot 7414 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-worker1142.eqiad.wmnet 2471686 wikitext text/x-wiki r21b8aa3ut98eg1uo9g19zaqu8mrj0y 2471687 2471686 2026-10-20T17:19:11Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471687 wikitext text/x-wiki dti8grb9du1s430lmk4t7ew2byunhrz 2471688 2471687 2026-10-20T17:20:43Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471688 wikitext text/x-wiki 1o6g7mntjsu9oxusn15b248ue8m9h92 2471689 2471688 2026-10-20T17:20:53Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T441633)', diff saved to https://phabricator.wikimedia.org/P96904 and previous config saved to /var/cache/conftool/dbconfig/20261020-172050-ladsgroup.json 2471689 wikitext text/x-wiki itzs0ftiqr7ur1p0t860p4d8iozzzb3 2471690 2471689 2026-10-20T17:21:10Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1187.eqiad.wmnet with reason: Maintenance 2471690 wikitext text/x-wiki d0fp12mkoo0s2ett5bd4mjx0pviu6r0 2471691 2471690 2026-10-20T17:21:15Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96905 and previous config saved to /var/cache/conftool/dbconfig/20261020-172110-ladsgroup.json 2471691 wikitext text/x-wiki py8ns2o5qfn67qyd8b5opsbezxpox15 2471692 2471691 2026-10-20T17:23:41Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1142.eqiad.wmnet 2471692 wikitext text/x-wiki ad1q0qun3n36yk5yzy1hb1qqezt3vn0 2471693 2471692 2026-10-20T17:26:07Z Stashbot 7414 blake@deploy1003: Started scap sync-world: bump chart version for T386694 2471693 wikitext text/x-wiki ld5u0vflxnv0lra69ckseln8ch0avle 2471694 2471693 2026-10-20T17:26:12Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet 2471694 wikitext text/x-wiki 0sdiuk16on699dqprlt9odkvjyz65zk 2471695 2471694 2026-10-20T17:28:05Z Stashbot 7414 mutante: re-enabled puppet on codesearch1001/2001 - might start to index git repos on gitlab and gerrit 2471695 wikitext text/x-wiki alpiysnpzz9qr4ub5q0h4vy361ur6n1 2471696 2471695 2026-10-20T17:28:10Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96906 and previous config saved to /var/cache/conftool/dbconfig/20261020-172807-ladsgroup.json 2471696 wikitext text/x-wiki bfl9n7rjvwlp0q83f4uj7t3cx538sck 2471697 2471696 2026-10-20T17:30:05Z Stashbot 7414 blake@deploy1003: Finished scap sync-world: bump chart version for T386694 (duration: 04m 54s) 2471697 wikitext text/x-wiki 9ve1qssttnrsarj9kuj60yikld0omfg 2471698 2471697 2026-10-20T17:38:11Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96908 and previous config saved to /var/cache/conftool/dbconfig/20261020-173809-ladsgroup.json 2471698 wikitext text/x-wiki gy4d21tuzbb6xfe3z3ctz36a7mohj2m 2471699 2471698 2026-10-20T17:39:30Z Stashbot 7414 brett@cumin2003: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir 2471699 wikitext text/x-wiki ipzuw48lpbhxcfmt3z295vj7yebcwkh 2471700 2471699 2026-10-20T17:41:00Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.302.0" for 3 host(s) 2471700 wikitext text/x-wiki bx1k89758fkif0ovk37msclhiv7j9vp 2471701 2471700 2026-10-20T17:41:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471701 wikitext text/x-wiki rhftzl76n9dyp70cjmo5ql58dzg40hg 2471702 2471701 2026-10-20T17:41:44Z Stashbot 7414 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad 2471702 wikitext text/x-wiki 3w2mybj7k02ocvqrgikndv3q82o5ghj 2471703 2471702 2026-10-20T17:42:55Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.302.0" completed for 3 hosts 2471703 wikitext text/x-wiki mt6zd8iyxzdg79w9j5fbazrvh8c68d4 2471704 2471703 2026-10-20T17:48:13Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P96909 and previous config saved to /var/cache/conftool/dbconfig/20261020-174811-ladsgroup.json 2471704 wikitext text/x-wiki rpkxy5ufg5rllzxededxzgnsf8uru0c 2471706 2471704 2026-10-20T17:58:15Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T441633)', diff saved to https://phabricator.wikimedia.org/P96910 and previous config saved to /var/cache/conftool/dbconfig/20261020-175812-ladsgroup.json 2471706 wikitext text/x-wiki 0utnhrq5y7bqaduj64s8shc3otldg8y 2471707 2471706 2026-10-20T17:58:33Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1201.eqiad.wmnet with reason: Maintenance 2471707 wikitext text/x-wiki k26ojudz5eg3rbe6twc36ywwh5v9kkd 2471708 2471707 2026-10-20T17:58:38Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96911 and previous config saved to /var/cache/conftool/dbconfig/20261020-175833-ladsgroup.json 2471708 wikitext text/x-wiki sglr6zv38e8ow0vplqxkjmk7sfh74yw 2471709 2471708 2026-10-20T18:05:39Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96912 and previous config saved to /var/cache/conftool/dbconfig/20261020-180536-ladsgroup.json 2471709 wikitext text/x-wiki 5svi5m8suspen5lbhjmu863dgmv2un4 2471710 2471709 2026-10-20T18:07:52Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet 2471710 wikitext text/x-wiki 6cx4s76pma7accnmzlg4q141dzypy50 2471711 2471710 2026-10-20T18:10:42Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471711 wikitext text/x-wiki 1xavz5y53gyv4nw16imfbgxf6c2zu71 2471712 2471711 2026-10-20T18:14:25Z Stashbot 7414 dancy@deploy1003: rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.1 refs T438222 2471712 wikitext text/x-wiki q4m5o48r9wc29qzim79dllxh0w0u6hi 2471713 2471712 2026-10-20T18:15:41Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96914 and previous config saved to /var/cache/conftool/dbconfig/20261020-181538-ladsgroup.json 2471713 wikitext text/x-wiki k5ekkbyjdths8y8hswzlljvo32se2a1 2471714 2471713 2026-10-20T18:24:22Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 2471714 wikitext text/x-wiki 6bsq20x1utvn45gnavhcqhjxvkd9wfn 2471715 2471714 2026-10-20T18:25:42Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96915 and previous config saved to /var/cache/conftool/dbconfig/20261020-182540-ladsgroup.json 2471715 wikitext text/x-wiki 0gx8w338hjw0bupwpvb7wb4ovbzjf8v 2471719 2471715 2026-10-20T18:35:44Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96916 and previous config saved to /var/cache/conftool/dbconfig/20261020-183542-ladsgroup.json 2471719 wikitext text/x-wiki 6m16ruhbaubembyef8f0c7rs6ncf29w 2471720 2471719 2026-10-20T18:36:03Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance 2471720 wikitext text/x-wiki 04y39sjlrtyuc8p7mvxy92psh1yi5mc 2471721 2471720 2026-10-20T18:41:35Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 2471721 wikitext text/x-wiki 73168glwwiznpkkkvy3f7xev9mbamxa 2471722 2471721 2026-10-20T18:48:09Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir 2471722 wikitext text/x-wiki 8j2osyg2mqqmk3ppagtkz3dxtotikb2 2471723 2471722 2026-10-20T18:48:41Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96918 and previous config saved to /var/cache/conftool/dbconfig/20261020-184838-ladsgroup.json 2471723 wikitext text/x-wiki tq0sfr7bvwadq7x30jdxx5szqasz1is 2471724 2471723 2026-10-20T18:49:09Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet 2471724 wikitext text/x-wiki iteduodbt6oj53isc55xvgra7vc8nyk 2471725 2471724 2026-10-20T18:58:43Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96919 and previous config saved to /var/cache/conftool/dbconfig/20261020-185840-ladsgroup.json 2471725 wikitext text/x-wiki 9x2khntn1k1nlcqo9ayvlqt1w3lnxgf 2471726 2471725 2026-10-20T19:01:10Z Stashbot 7414 dancy@deploy1003: Installing scap version "4.302.0" for 158 host(s) 2471726 wikitext text/x-wiki jgc36mewk1804qzjl8wyq2dyzmsrnmx 2471727 2471726 2026-10-20T19:02:37Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 2471727 wikitext text/x-wiki oc5z4svslegi14bpcry5ovg8z7qfxwe 2471730 2471727 2026-10-20T19:03:47Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471730 wikitext text/x-wiki spkwb2sn7z4bstnscc8v3gfhz1rpjwm 2471731 2471730 2026-10-20T19:03:51Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471731 wikitext text/x-wiki 7zv0ck2s80hqlsnuj6y8tkdvjekyf3v 2471732 2471731 2026-10-20T19:05:14Z Stashbot 7414 dancy@deploy1003: Installation of scap version "4.302.0" completed for 158 hosts 2471732 wikitext text/x-wiki 1lx3dmqpnk0i4oeb746m8qhvdegfkik 2471733 2471732 2026-10-20T19:05:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471733 wikitext text/x-wiki 5z1bica34jx9a5ssp09uu32mkohfvbt 2471734 2471733 2026-10-20T19:07:01Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 2471734 wikitext text/x-wiki 9fmz94lo2u5mb1ofcotjcyzr30xnu88 2471735 2471734 2026-10-20T19:08:45Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96920 and previous config saved to /var/cache/conftool/dbconfig/20261020-190842-ladsgroup.json 2471735 wikitext text/x-wiki hk5xyju5rzmqj9ofsoyrl2yi53a9oqj 2471736 2471735 2026-10-20T19:18:47Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96921 and previous config saved to /var/cache/conftool/dbconfig/20261020-191844-ladsgroup.json 2471736 wikitext text/x-wiki royqpnc47dc7oih12ljw3mw34y93c7t 2471737 2471736 2026-10-20T19:19:05Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1279.eqiad.wmnet with reason: Maintenance 2471737 wikitext text/x-wiki 4egq143y18plop8iejfaknq2x2vol5s 2471738 2471737 2026-10-20T19:19:26Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance 2471738 wikitext text/x-wiki ffusmbg2yqlgsl57r48227bm8n15axe 2471739 2471738 2026-10-20T19:19:30Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96922 and previous config saved to /var/cache/conftool/dbconfig/20261020-191925-ladsgroup.json 2471739 wikitext text/x-wiki r08fux8dteqnlh7zdcwuhomkgjqmqmb 2471740 2471739 2026-10-20T19:23:23Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471740 wikitext text/x-wiki h3pc9vhsg8ag7eai9izf088rqugz1fu 2471741 2471740 2026-10-20T19:24:28Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96923 and previous config saved to /var/cache/conftool/dbconfig/20261020-192424-ladsgroup.json 2471741 wikitext text/x-wiki 4i35nxn7lx2i68ihpdfsdf8wg3251zw 2471744 2471741 2026-10-20T19:28:46Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet 2471744 wikitext text/x-wiki h2jfnyaqe8wk67ucqrhiofwc0kybqc7 2471746 2471744 2026-10-20T19:34:29Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96924 and previous config saved to /var/cache/conftool/dbconfig/20261020-193426-ladsgroup.json 2471746 wikitext text/x-wiki h25vy7lnw5vp4sdt8h06fx6jmqdoade 2471747 2471746 2026-10-20T19:40:36Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471747 wikitext text/x-wiki ea8viai634ed7gk6md92nv0rzik321d 2471748 2471747 2026-10-20T19:44:31Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96925 and previous config saved to /var/cache/conftool/dbconfig/20261020-194428-ladsgroup.json 2471748 wikitext text/x-wiki a7tyh05xpmw7h8cvm2x1wj450yb73fs 2471749 2471748 2026-10-20T19:54:33Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96926 and previous config saved to /var/cache/conftool/dbconfig/20261020-195430-ladsgroup.json 2471749 wikitext text/x-wiki 5qeb9owgnhl2e04zp699m9byx98c3mr 2471750 2471749 2026-10-20T19:54:51Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1282.eqiad.wmnet with reason: Maintenance 2471750 wikitext text/x-wiki sf0wbugwtjzx5lsp9x3o5ikf88um452 2471751 2471750 2026-10-20T19:54:56Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96927 and previous config saved to /var/cache/conftool/dbconfig/20261020-195450-ladsgroup.json 2471751 wikitext text/x-wiki pwq4iqizg36k2uoo8pcn49tpqek7my9 2471752 2471751 2026-10-20T19:58:10Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471752 wikitext text/x-wiki 6a3108dkoiia3k87q8ou8xdhbduy04i 2471753 2471752 2026-10-20T19:59:53Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96928 and previous config saved to /var/cache/conftool/dbconfig/20261020-195951-ladsgroup.json 2471753 wikitext text/x-wiki ltfc6il555pt94qb4n7uaq9jjfwwe7j 2471755 2471753 2026-10-20T20:06:35Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) 2471755 wikitext text/x-wiki sbo9nti6ukhgxprrbgc97o0lyidd54m 2471756 2471755 2026-10-20T20:08:15Z Stashbot 7414 brett@cumin2003: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet 2471756 wikitext text/x-wiki hh7gdme0r9dhmqxdt0hw72hniufr0y9 2471757 2471756 2026-10-20T20:08:19Z Stashbot 7414 brett@cumin2003: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp 2471757 wikitext text/x-wiki olm09u6u5p1pjpptsivtk23ca22zakr 2471759 2471757 2026-10-20T20:09:55Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96929 and previous config saved to /var/cache/conftool/dbconfig/20261020-200953-ladsgroup.json 2471759 wikitext text/x-wiki 84e98j0yyz1u6y0t63f6q2phazaeeh5 2471760 2471759 2026-10-20T20:10:49Z Stashbot 7414 arlolra@deploy1003: arlolra: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2471760 wikitext text/x-wiki 6imy2f3sch1oiiz1596zyedreolcd9o 2471761 2471760 2026-10-20T20:11:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 2471761 wikitext text/x-wiki rgd6xh2a5270t9kh1mme4ci3b79bwc7 2471762 2471761 2026-10-20T20:11:21Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 2471762 wikitext text/x-wiki l3smlvfr17g82yu2hsf1ohiozqzun2g 2471763 2471762 2026-10-20T20:11:50Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 2471763 wikitext text/x-wiki 8unrqqmmvx72egu1efrxth5cmcqybc5 2471764 2471763 2026-10-20T20:11:58Z Stashbot 7414 arlolra@deploy1003: arlolra: Continuing with deployment 2471764 wikitext text/x-wiki am4s2i51xxulzmg1o9v7f39ghygzhcu 2471766 2471764 2026-10-20T20:19:57Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96930 and previous config saved to /var/cache/conftool/dbconfig/20261020-201954-ladsgroup.json 2471766 wikitext text/x-wiki ceyntenl2k6kjhu4kcwjrf8oirq7zvq 2471767 2471766 2026-10-20T20:25:33Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) (duration: 18m 59s) 2471767 wikitext text/x-wiki 1p35tey4v9pn6f9upc77g5nrnr6bp1w 2471768 2471767 2026-10-20T20:29:16Z Stashbot 7414 arlolra@deploy1003: Started scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] 2471768 wikitext text/x-wiki e9j892bi3o7xg78ra9kh4xtxf5bbz5u 2471769 2471768 2026-10-20T20:29:59Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96931 and previous config saved to /var/cache/conftool/dbconfig/20261020-202957-ladsgroup.json 2471769 wikitext text/x-wiki ftar42ow610dr9zdzdc2cphh0430kqc 2471770 2471769 2026-10-20T20:30:17Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1287.eqiad.wmnet with reason: Maintenance 2471770 wikitext text/x-wiki owhq52cnm4rfzomf4j3ns8gd031hdiw 2471771 2471770 2026-10-20T20:30:22Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96932 and previous config saved to /var/cache/conftool/dbconfig/20261020-203016-ladsgroup.json 2471771 wikitext text/x-wiki g9yx98zxv8qun80ayyo0xce5yd0wiu3 2471772 2471771 2026-10-20T20:30:52Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet 2471772 wikitext text/x-wiki 61qn8oqhz49q3404rc21clhw6hvysw2 2471773 2471772 2026-10-20T20:33:26Z Stashbot 7414 arlolra@deploy1003: ssastry, arlolra: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2471773 wikitext text/x-wiki fuhpm560p4mzmb55jkjm409q2cj3iyu 2471774 2471773 2026-10-20T20:34:03Z Stashbot 7414 arlolra@deploy1003: ssastry, arlolra: Continuing with deployment 2471774 wikitext text/x-wiki aj182hr2b0tk23gjgqrk7ve8as47gne 2471775 2471774 2026-10-20T20:35:23Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96933 and previous config saved to /var/cache/conftool/dbconfig/20261020-203520-ladsgroup.json 2471775 wikitext text/x-wiki 7v4yijlcalg6pq0u17pa4gxy5yd268u 2471776 2471775 2026-10-20T20:38:01Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet 2471776 wikitext text/x-wiki cegsxuz41e2u914c79unh2bj5gblmv8 2471777 2471776 2026-10-20T20:39:45Z Stashbot 7414 arlolra@deploy1003: Finished scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] (duration: 10m 37s) 2471777 wikitext text/x-wiki m0a5mn1cr7fgnuu885s6nnglpplzgid 2471778 2471777 2026-10-20T20:45:25Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96934 and previous config saved to /var/cache/conftool/dbconfig/20261020-204522-ladsgroup.json 2471778 wikitext text/x-wiki qpr65ojha1tdvoahk6dnts7mu337e91 2471779 2471778 2026-10-20T20:53:14Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471779 wikitext text/x-wiki 4ibeztflw0k8yltfov4sg1iq16pvjk0 2471780 2471779 2026-10-20T20:53:19Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471780 wikitext text/x-wiki 5mr77ocn8wzgugon3p43iguvoebv3bz 2471781 2471780 2026-10-20T20:53:28Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471781 wikitext text/x-wiki c2uflx74lmosbp37akamcbl8p8d0lb3 2471782 2471781 2026-10-20T20:55:26Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96935 and previous config saved to /var/cache/conftool/dbconfig/20261020-205524-ladsgroup.json 2471782 wikitext text/x-wiki pwst0wrbzu5qv07bgjypmothgg16w35 2471784 2471782 2026-10-20T21:05:29Z Stashbot 7414 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96937 and previous config saved to /var/cache/conftool/dbconfig/20261020-210526-ladsgroup.json 2471784 wikitext text/x-wiki e7zsg679ue8t16h4rwq1sahx57zjfvu 2471785 2471784 2026-10-20T21:05:46Z Stashbot 7414 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance 2471785 wikitext text/x-wiki 4xp3nxxyc13mppg3j2fh3c6293xpki6 2471786 2471785 2026-10-20T21:14:40Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471786 wikitext text/x-wiki 5u39x5b58mgp5dpz1gafcthgvsvu2tw 2471788 2471786 2026-10-20T21:24:59Z Stashbot 7414 thcipriani@deploy1003: Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week 2471788 wikitext text/x-wiki kzebkk9s9t73y32ugsz0xtllznyjno1 2471789 2471788 2026-10-20T21:39:10Z Stashbot 7414 bking@cumin2003: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 2471789 wikitext text/x-wiki f1m4v240a7k6m9e7kndxivu6acbcq77 2471791 2471789 2026-10-20T21:41:57Z Stashbot 7414 bking@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudelastic1008.eqiad.wmnet with reason: backplane error 2471791 wikitext text/x-wiki o4iz6cimlv0wgzkjqr0d7xiymcmmkdz 2471793 2471791 2026-10-20T21:49:11Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471793 wikitext text/x-wiki tfsq5nixg47dfbohy5rebkay6lmleup 2471794 2471793 2026-10-20T21:49:16Z Stashbot 7414 bking@cumin2003: END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) 2471794 wikitext text/x-wiki 8ttewzamt3bw2a10fre7ad28yfikjqm 2471795 2471794 2026-10-20T21:51:59Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.wdqs.reboot 2471795 wikitext text/x-wiki s8zv3prjdl8p56t075lf78cjyorcnln 2471796 2471795 2026-10-20T21:55:01Z Stashbot 7414 bking@cumin2003: START - Cookbook sre.hosts.reboot-single for host wdqs1027.eqiad.wmnet 2471796 wikitext text/x-wiki djyke37t8xxf4sktix7tyhdfofgsq05 2471797 2471796 2026-10-20T22:02:15Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1027.eqiad.wmnet 2471797 wikitext text/x-wiki aga2wtvo4tyg2u9o210bo9yq6jqkdxu 2471798 2471797 2026-10-20T22:10:36Z Stashbot 7414 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) 2471798 wikitext text/x-wiki q87f8vcvmootc26qpf3g3o47ly3gguj 2471801 2471798 2026-10-20T23:11:50Z Stashbot 7414 thcipriani@deploy1003: Forcefully removing global lock: Pausing deploy lock for quick backport 2471801 wikitext text/x-wiki 5adauiythb02p4adckkqnwj708p08f2 2471802 2471801 2026-10-20T23:12:03Z Stashbot 7414 thcipriani@deploy1003: Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 107m 04s) 2471802 wikitext text/x-wiki sbqkysjzqmicgpi74jwzuaacaepmene 2471803 2471802 2026-10-20T23:15:20Z Stashbot 7414 thcipriani@deploy1003: Started scap sync-world: Backport for PhpVersion: Show only the major and minor PHP version ([[gerrit:1357432|1.47.0-wmf.24]], [[gerrit:1357431|1.48.0-wmf.1]]) 2471803 wikitext text/x-wiki nfjmzgt5edzg68sllt4na2aelkcfh2e 2471804 2471803 2026-10-20T23:19:36Z Stashbot 7414 thcipriani@deploy1003: thcipriani: Backport for PhpVersion: Show only the major and minor PHP version ([[gerrit:1357432|1.47.0-wmf.24]], [[gerrit:1357431|1.48.0-wmf.1]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2471804 wikitext text/x-wiki p899x6z5h9fis18f2s0984wn38djiei 0 13333 2471820 2449833 2026-10-21T08:15:31Z Hashar 32 Add navigation template [[template:Navigation Wikimedia infrastructure]] 2471820 wikitext text/x-wiki 7nq1gnen8ema80oyf4xu5istuqqv3qe 0 17290 2471758 2470809 2026-10-20T20:08:27Z Stashbot 7414 Southparkfan: Remove cherry-pick for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1341351 to fix merge conflict - T441851 2471758 wikitext text/x-wiki 6cqva59p3glsumslh657h55tukc143z 2471783 2471758 2026-10-20T21:05:18Z Stashbot 7414 Southparkfan: Beta Cluster: allow 873/tcp to swift-fe security group to fix fetch-rings-deployment-prep.service on Puppetserver - T436476 2471783 wikitext text/x-wiki fv8snd4awy32edf5q80a99i2bq17an3 2471792 2471783 2026-10-20T21:48:51Z Stashbot 7414 Southparkfan: Beta Cluster: Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1357408 - T436476 2471792 wikitext text/x-wiki e3sllsby55wnp5bazpti9agpmzigcug 2471799 2471792 2026-10-20T22:59:32Z Stashbot 7414 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/125 (T438720) 2471799 wikitext text/x-wiki 5ap1ezuh88dpsop1xvjl9f12bjxcfaz 498 30942 2471425 2471367 2026-10-20T12:41:41Z Stashbot 7414 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudweb.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' 2471425 wikitext text/x-wiki 1n9npv2666jsvdu947e309y47l051bv 2471429 2471425 2026-10-20T12:46:36Z Stashbot 7414 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudweb.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::cloudweb}' 2471429 wikitext text/x-wiki fsva3zwpf4az0i0jvbb4rqh6mn4z7mh 2471612 2471429 2026-10-20T15:45:25Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_mons (T441652) 2471612 wikitext text/x-wiki 04bdmvqa8hd515sc6qxhqyoyfg9uqeu 2471635 2471612 2026-10-20T16:14:09Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_mons (exit_code=0) (T441652) 2471635 wikitext text/x-wiki 4hvwo0ekoqjp5zalcfs2yhdjp7wifd0 2471636 2471635 2026-10-20T16:15:11Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.ceph.roll_reboot_osds (T441652) 2471636 wikitext text/x-wiki t25hp62mxk8ihvdse2po72nlhzbop80 2471716 2471636 2026-10-20T18:29:00Z Stashbot 7414 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.cloudservices.safe_reboot on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' 2471716 wikitext text/x-wiki 0w110vjkuk6ajkf0bstrnzhu3s002gh 2471717 2471716 2026-10-20T18:35:15Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudservices.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::eqiad1::services}' 2471717 wikitext text/x-wiki 4votu6zg19btv84kyx7q4hyzgbecv9z 2471742 2471717 2026-10-20T19:24:40Z Stashbot 7414 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.ceph.roll_reboot_osds (exit_code=0) (T441652) 2471742 wikitext text/x-wiki ts4d6eogs7g2ovcs2g67gnpcw2ekqwb 10 37907 2471817 2438594 2026-10-21T08:11:51Z Hashar 32 Add Aphlict, a component of Phabricator 2471817 wikitext text/x-wiki 3ktp5w5qm0xmsyrp3cpoje3897vzp6j 2471819 2471817 2026-10-21T08:14:39Z Hashar 32 Add doc.wikimedia.org and Zuul 2471819 wikitext text/x-wiki 9hbwt113qx4lvy7k7v9tvx5f6l9686e 8 144571 2471809 2396034 2026-10-21T00:25:41Z TCipriani (WMF) 4895 2471809 wikitext text/x-wiki 7wjbjoqy0m8bafqyauk7exkkhousmok 0 440900 2471392 2471376 2026-10-20T12:06:53Z Filippo Giunchedi 2939 /* cloudvirtlocalXXXX */ Add k8s dashboard link for etcd health 2471392 wikitext text/x-wiki 6u48zyrsnqsadsi859b4s1drigzhw3e 2471443 2471392 2026-10-20T13:12:06Z Filippo Giunchedi 2939 /* cloudrabbitXXXX */ Link dashboard 2471443 wikitext text/x-wiki quxtid26en3ftw2evn1y75f4f0bxyp4 0 443740 2471424 2465897 2026-10-20T12:41:25Z Ottomata 88 eventgate-analytics refetches stream config since https://phabricator.wikimedia.org/T430154 2471424 wikitext text/x-wiki tmsing0zi5ojsoovl8avyizxfhtr0gn 0 444148 2471827 2463378 2026-10-21T11:19:41Z AChou-WMF 15621 Update MI300X-partitions diagram 2471827 wikitext text/x-wiki j0sw2gjlu19rkxbi8l07m2fj7s9g01u 2471829 2471827 2026-10-21T11:35:30Z AChou-WMF 15621 Updated outdated content 2471829 wikitext text/x-wiki nksfygdx5ojwimdqllrqy24872ms87s 116 444352 2471805 2214862 2026-10-20T23:36:13Z Diegodlh 29411 Added comment that Telegram bot privacy had to be disabled, for reproducibility 2471805 wikitext text/x-wiki p59i176vawqq05ptspaz1milqcstou5 0 447260 2471818 2411613 2026-10-21T08:13:17Z LToscano (WMF) 7191 /* Renewing an existing intermediate */ 2471818 wikitext text/x-wiki e7e0r90vj5fyoskiot921b34qnrji81 0 447389 2471821 2466270 2026-10-21T08:16:18Z LToscano (WMF) 7191 2471821 wikitext text/x-wiki q2ioweq0a9bk4wfe1w38kpdf34xpols 12 449113 2471705 2463576 2026-10-20T17:55:19Z Owuh 39223 /* Deleting your jobs */ extra space 2471705 wikitext text/x-wiki 748ck816fxispu0rk24lgtsu50bvl37 0 449160 2471629 2471179 2026-10-20T16:07:02Z Dexbot 30554 Bot: Updating the report 2471629 wikitext text/x-wiki lcbw4v298zlvnsxm32kqtmuh2y4gt56 0 451477 2471816 2076308 2026-10-21T08:10:50Z Hashar 32 Add navigation template [[template:Navigation Wikimedia infrastructure]] 2471816 wikitext text/x-wiki pl3v8jol2l7592i55a496rcb07q9st4 0 453382 2471411 2471260 2026-10-20T12:22:34Z CMooney (WMF) 24288 /* Troubleshooting */ 2471411 wikitext text/x-wiki r2du33qspsldlhj2zanvc8vrmlmqpik 2471412 2471411 2026-10-20T12:23:05Z CMooney (WMF) 24288 /* Restart subscription for a given device */ 2471412 wikitext text/x-wiki cbewmmpp0gud2ib5t48j2xirgtgw1bn 2471422 2471412 2026-10-20T12:38:25Z CMooney (WMF) 24288 /* Restart subscription for a given device */ 2471422 wikitext text/x-wiki 0zy9txmkcrc6dh6qlrfgn9wcql9uqgp 2471423 2471422 2026-10-20T12:38:47Z CMooney (WMF) 24288 /* View subscription status for a given device */ 2471423 wikitext text/x-wiki chmdtz7tkytxr1r8uvfe9vqmwfynetd 0 453597 2471638 2463455 2026-10-20T16:25:50Z FNegri-WMF 32595 /* 2026 */ Add rotation of db credentials 2471638 wikitext text/x-wiki qlmfkb83tl4r3tybxxw580rqwhpbjyu 116 453906 2471498 2471267 2026-10-20T14:00:50Z Gitlabaccountapprovalbot 37332 @eudaimonia1 was approved. 2471498 wikitext text/x-wiki miift26bjkuwi6q9lyt3kpg04eudkm7 2471525 2471498 2026-10-20T14:15:43Z Gitlabaccountapprovalbot 37332 @ryuch was approved. 2471525 wikitext text/x-wiki nv0emz7e3kf0ibp6oyj0but76rl5vh0 0 455357 2471823 2454902 2026-10-21T09:07:15Z Effie Mouzeli (WMF) 12880 2471823 wikitext text/x-wiki lxax3wso9hqx5udn8mww23jzz26ovvj 498 458744 2471745 2325027 2026-10-20T19:31:24Z Stashbot 7414 wmbot~bd808@tools-bastion-14: Restart to pick up new container built from 86606dbd (T441864) 2471745 wikitext text/x-wiki lzxgi8m6oyk3ua1iahgt15d2rau8e4d 0 459290 2471549 2470907 2026-10-20T14:27:04Z APizzata-WMF 47941 tuesday train 2471549 wikitext text/x-wiki l5rtjjbpgqjatg05zes0dfwjf89jbrh 0 459699 2471754 2471011 2026-10-20T20:05:32Z Stashbot 7414 jgleeson: payments-wiki upgraded from cced0410 to c32d227e 2471754 wikitext text/x-wiki d5a8165pnext15gjzxh6yq43hzr47g8 2471787 2471754 2026-10-20T21:18:38Z Stashbot 7414 eileen: civicrm upgraded from 348449da to 6ff93d7a 2471787 wikitext text/x-wiki a6rleryes2m6aju51tsmd3vqalrg6mb 2471800 2471787 2026-10-20T23:03:50Z Stashbot 7414 ejegg: donorwiki upgraded from 06bf0472 to c32d227e 2471800 wikitext text/x-wiki a8vkjeyxxgtwshkudthoe018x7lk522 0 460289 2471488 2426014 2026-10-20T13:52:50Z AKlapper (WMF) 421 correct branching command 2471488 wikitext text/x-wiki siy8u694u8imivecxluf0zl5sbak7f4 2471547 2471488 2026-10-20T14:26:21Z AKlapper (WMF) 421 2471547 wikitext text/x-wiki cjd8yy6j8bhpjavqrkw6ksuee407ba4 2471610 2471547 2026-10-20T15:43:55Z AKlapper (WMF) 421 2471610 wikitext text/x-wiki 8xq6cd09nhn221optf7sr555rm8si2j 0 460417 2471815 2470795 2026-10-21T05:04:57Z ISarantopoulos-WMF 34784 Undid revision [[Special:Diff/2470795|2470795]] by [[Special:Contributions/Cheaterblocker|Cheaterblocker]] ([[User talk:Cheaterblocker|talk]]) 2471815 wikitext text/x-wiki stjebalzfapfjf9iajpa8yychiitbgm 0 460515 2471718 2469943 2026-10-20T18:35:20Z Ottomata 88 /* Prefetches */ linked high_traffic_jobs to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/services/changeprop-jobqueue/values.yaml#31 2471718 wikitext text/x-wiki la1pvszv5ssr35qfck8jepf4h6z4v8w 0 460789 2471824 2470504 2026-10-21T10:19:46Z GKyziridis-WMF 39160 Remove the gpt-oss model 2471824 wikitext text/x-wiki jy56ifb1b7m59w9241ohjy9lky8w5ta 2471828 2471824 2026-10-21T11:25:57Z GKyziridis-WMF 39160 Remove qwen36 since we will going to deprecate it and keep only qwen38 2471828 wikitext text/x-wiki hepl9r6ftmdlwl67i0waje3sgxmxp42 0 460832 2471390 2471380 2026-10-20T12:05:17Z OKarakaya-WMF 40765 clean up 2471390 wikitext text/x-wiki cf31ffw5m3vzie8ce12bhmionzg01xb 2471397 2471390 2026-10-20T12:10:18Z OKarakaya-WMF 40765 update dashboards 2471397 wikitext text/x-wiki qcsw760v0hreqbs9ojbiqsfyqgnihxk 2471822 2471397 2026-10-21T08:49:06Z OKarakaya-WMF 40765 clean up 2471822 wikitext text/x-wiki i7nyqplsjefkj45eyhdsvj0rka6hujt 3 460833 2471472 2026-10-20T13:42:11Z StrikerBot 8475 Welcome to Toolforge! 2471472 wikitext text/x-wiki fwhsm66710uunz1ftgn4hrf8c424iag 3 460834 2471479 2026-10-20T13:46:50Z StrikerBot 8475 Welcome to Toolforge! 2471479 wikitext text/x-wiki mhwpwo6e58ytc9gx7o1uj803uwai3m3 3 460835 2471486 2026-10-20T13:51:57Z StrikerBot 8475 Welcome to Toolforge! 2471486 wikitext text/x-wiki mrhh6boceoq3x9k79h093293n21v2o5 3 460836 2471505 2026-10-20T14:06:57Z StrikerBot 8475 Welcome to Toolforge! 2471505 wikitext text/x-wiki hgt6guahh09tgm0gn56tm2j1txvbzwv 3 460837 2471536 2026-10-20T14:19:49Z StrikerBot 8475 Welcome to Toolforge! 2471536 wikitext text/x-wiki ercc2hzq8hyvvdoyree81jtvwmj1wi2 0 460839 2471806 2026-10-21T00:01:07Z KHarlan (WMF) 12566 Created page with "{{irdoc|status=draft}} == Summary == {{Incident scorecard | task = | paged-num = | responders-num = | coordinators = | start = | end = }} … <!-- Reminder: No private information on this page! --><mark>Summary of what happened, in one or two paragraphs. Avoid assuming deep knowledge of the systems here, and try to differentiate between proximate causes and root causes.</mark> {{TOC|align=right}} ==Timeline== <mark>Write a step by step outline of what happened..." 2471806 wikitext text/x-wiki tn08wk76xcoau44g55y7bsgo9lps8zo 0 460840 2471807 2026-10-21T00:16:58Z AndrewBogott (WMF) 51047 Created page with "Events are developing rapidly. This page will be updated with updates as appropriate. Currently, all of cloud-vps and toolforge is offline and is likely to remain so through late October." 2471807 wikitext text/x-wiki eisfpovmgi4osdc0yzlt6i2a462ibw4 2471808 2471807 2026-10-21T00:17:24Z AndrewBogott (WMF) 51047 2471808 wikitext text/x-wiki jxssqynrhargmp8swmlro95xomr4r1s 2471810 2471808 2026-10-21T00:48:32Z BPiovesan-WMF 56466 Updates sites that are currently offline 2471810 wikitext text/x-wiki tg4syq6rjmlcacdf54b7ccbjymth5a6 2471811 2471810 2026-10-21T00:50:27Z TCipriani (WMF) 4895 2471811 wikitext text/x-wiki kxk0nvjc7inmdjmkpvl8aepht59l40k 2471812 2471811 2026-10-21T01:19:08Z JJMC89 7474 more WMCS domains for clarity 2471812 wikitext text/x-wiki 0s9dhqv42qosvgmygzz9io7lxr6184o 2471813 2471812 2026-10-21T01:34:36Z BPiovesan-WMF 56466 Update to the services that are down 2471813 wikitext text/x-wiki 5vcnp5x78ikxzj8dnvozi9likfan2eq 2471814 2471813 2026-10-21T04:13:40Z APatro (WMF) 38793 2471814 wikitext text/x-wiki llanqp79nv65hqqxrf3ciz2ovi8kok0 2471825 2471814 2026-10-21T10:28:43Z KHarlan (WMF) 12566 2471825 wikitext text/x-wiki e9h4bsruk493fmhf2moc5hml8bm7atb 2471826 2471825 2026-10-21T10:40:28Z RSilvola-WMF 49112 Add recommendation to MW operators 2471826 wikitext text/x-wiki 3riypi4yw8rt9d26fkqmei4b116m163