Wikitech labswiki https://wikitech.wikimedia.org/wiki/Main_Page MediaWiki 1.47.0-wmf.17 first-letter Media Special Talk User User talk Wikitech Wikitech talk File File talk MediaWiki MediaWiki talk Template Template talk Help Help talk Category Category talk Obsolete Obsolete talk OfficeIT OfficeIT talk Tool Tool talk Nova Resource Nova Resource Talk Heira Heira Talk TimedText TimedText talk Module Module talk 498 3086 2452778 2452286 2026-08-30T20:51:25Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2452778 wikitext text/x-wiki === 2026-08-30 === * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:26 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:43 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:17 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:15 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:03 dhinus: restart harbor in tools-harbor-2 with "docker compose restart" === 2026-08-26 === * 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 10:00 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:17 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-24 === * 10:46 taavi: disable access to elastic cluster as announced on cloud-announce [[phab:T401818|T401818]] === 2026-08-20 === * 16:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:41 dcaro: tools-puppetserver-01 was out of memory because of java and killed sssd backend * 15:33 dcaro: restart tools-puppetserver-01, unresponsive * 14:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:38 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:29 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:48 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 13:42 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:03 dcaro: truncate haproxy.log to 1G (out of space) on haproxy-8 === 2026-08-14 === * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 * 10:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 * 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 === 2026-08-13 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 11:25 taavi: update striker to 2026-08-13-111038-production * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 09:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2026-08-12 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 13:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-05 === * 14:43 taavi: update striker to 2026-08-04-183239-production === 2026-08-04 === * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:22 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:15 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-03 === * 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, t * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 17:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers ([[phab:T433426|T433426]]) * 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T433426|T433426]]) * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-31 === * 13:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:03 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:36 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:28 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:18 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 15:10 godog: hard reboot tools-bastion-15 * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:05 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:04 dcaro: reboot tools-bastion-15, getting out of memory * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:00 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:29 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 09:24 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 09:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 09:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 09:16 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 * 09:16 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 === 2026-07-29 === * 09:54 godog: test dumps-nfs failover - [[phab:T432325|T432325]] * 09:48 godog: enable 'dumps_use_nfs_lb: true' for nfs worker prefix - [[phab:T432325|T432325]] * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 08:56 godog: enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - [[phab:T432325|T432325]] === 2026-07-28 === * 08:25 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, t * 06:56 godog: resume nfs workers roll restart - [[phab:T432325|T432325]] * 06:55 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs === 2026-07-27 === * 17:18 dcaro: increase object storage object number quota to 1M ([[phab:T433281|T433281]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:06 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 11:39 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 11:37 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) * 08:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) === 2026-07-23 === * 17:11 dcaro: resize haproxy-8 * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:55 dcaro: resize haproxy-7 * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram * 12:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-07-20 === * 12:28 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 12:22 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:43 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:33 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-16 === * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:38 taavi: failing over redis to new trixie cluster [[phab:T360704|T360704]] * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 * 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 * 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 * 01:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 19:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 19:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 18:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' * 18:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud * 13:49 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:00 dcaro: restart haproxy on haproxy-8 removing tune.disable-zero-copy-forwarding from config === 2026-07-12 === * 21:01 dcaro: restart haproxy on haproxy-8 * 06:28 dcaro: restart haproxy on haproxy-8 === 2026-07-11 === * 17:34 dcaro: rebooting haproxy-8 === 2026-07-09 === * 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:27 taavi: rebooting bastions for kernel upgrades * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) === 2026-06-24 === * 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-15 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]] * 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]] === 2026-06-11 === * 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-09 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]] === 2026-06-04 === * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry === 2026-06-02 === * 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]] * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 20:33 andrewbogott: rebuilt pywikibot for v11.3.0 [[phab:T426547|T426547]] * 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]] * 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]] === 2026-05-21 === * 12:46 taavi: renew prometheus certificate * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-05-20 === * 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-16 === * 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers === 2026-05-15 === * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates === 2026-05-14 === * 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-13 === * 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]] * 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]] * 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2026-05-11 === * 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-07 === * 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]] === 2026-05-05 === * 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed * 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]] * 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]] === 2026-04-21 === * 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]] * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-16 === * 13:09 taavi: bump istio traffic percentage 75% -> 100% [[phab:T392356|T392356]] === 2026-04-15 === * 10:45 taavi: bump istio traffic percentage 50% -> 75% [[phab:T392356|T392356]] === 2026-04-13 === * 09:11 taavi: bump istio traffic percentage 25% -> 50% [[phab:T392356|T392356]] * 07:33 taavi: bump istio traffic percentage 10% -> 25% [[phab:T392356|T392356]] === 2026-04-10 === * 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-04-07 === * 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-30 === * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 dcaro: removing wal from prometheus nodes to restart them === 2026-03-26 === * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-25 === * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04 * 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04 === 2026-03-24 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]] * 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]] * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest * 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest * 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest * 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest * 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest === 2026-03-18 === * 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]] * 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster * 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]]) === 2026-03-17 === * 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]] === 2026-03-15 === * 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR === 2026-03-13 === * 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]] * 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]] === 2026-03-12 === * 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2026-03-10 === * 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster === 2026-03-06 === * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-03-05 === * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-03-04 === * 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97) * 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]]) * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2026-03-03 === * 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-25 === * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space * 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 === 2026-02-24 === * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-20 === * 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} === 2026-02-11 === * 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]] === 2026-02-05 === * 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs === 2026-02-04 === * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 * 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 === 2026-02-03 === * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 === 2026-01-28 === * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-14 === * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) * 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) === 2026-01-12 === * 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 200k * 11:05 godog: bump object quota to 500G === 2025-12-17 === * 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) === 2025-12-15 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]]) === 2025-12-14 === * 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert === 2025-12-11 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) === 2025-12-03 === * 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) === 2025-12-02 === * 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch * 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining * 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]]) === 2025-12-01 === * 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-27 === * 12:15 volans: [continue] on the haproxy nodes * 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}} === 2025-11-26 === * 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts * 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]] === 2025-11-25 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 * 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-19 === * 16:19 andrewbogott: increased object count quota to 100,000 * 16:03 andrewbogott: increased object storage quota to 200GB * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-11-18 === * 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1) * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1) * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81 * 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81 * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80 * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80 * 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1) * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1) * 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1) === 2025-11-17 === * 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]]) * 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]]) === 2025-11-14 === * 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]]) * 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]]) === 2025-11-13 === * 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1) * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1) * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1) === 2025-11-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-11 === * 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]]) === 2025-11-10 === * 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-07 === * 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]]) * 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]]) === 2025-11-06 === * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian * 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) === 2025-11-04 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-10-30 === * 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]] * 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]]) * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6 * 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6 * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-10-27 === * 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]] * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-24 === * 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t * 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k * 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106 * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106 * 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103 * 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]] === 2025-10-23 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]] * 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3 * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]] * 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]] === 2025-10-21 === * 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]] * 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]] * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-20 === * 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]] === 2025-10-16 === * 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-15 === * 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]] === 2025-10-14 === * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3 === 2025-10-13 === * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]] * 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 === 2025-10-10 === * 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 * 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 === 2025-10-09 === * 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2025-10-08 === * 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]] * 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 * 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 === 2025-10-07 === * 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11 * 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69 * 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69 * 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-06 === * 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7 * 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7 * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 * 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 === 2025-10-03 === * 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6 * 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6 * 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5 * 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5 === 2025-10-02 === * 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 * 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 === 2025-10-01 === * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-30 === * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 * 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 === 2025-09-29 === * 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 07:00 godog: kick stuck nfs workers from clouddumps1001 === 2025-09-28 === * 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-25 === * 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 13:49 dcaro: patched all tools with new resource defaults, everything looks good * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs * 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 === 2025-09-23 === * 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues * 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-09-22 === * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-21 === * 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page * 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster * 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts * 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-09-19 === * 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 === 2025-09-18 === * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]] === 2025-09-17 === * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 * 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 === 2025-09-16 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]] * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]] * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13 * 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13 * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12 * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12 * 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-15 === * 16:22 taavi: reboot old bastions to kick long-living connections into newer ones * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 * 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 === 2025-09-12 === * 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]] * 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]] * 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday === 2025-09-11 === * 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]] * 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own * 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state * 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-09-10 === * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) === 2025-09-09 === * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]]) * 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]]) === 2025-09-08 === * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests === 2025-09-06 === * 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 * 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 === 2025-09-05 === * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf === 2025-09-03 === * 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-29 === * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting === 2025-08-28 === * 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]] * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:05 taavi: copy missing aptly packages to trixie-<nowiki>{</nowiki>tools,toolsbeta<nowiki>}</nowiki> * 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-26 === * 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]]) * 10:25 dhinus: shut down tools-harbor-1 (no longer used) === 2025-08-25 === * 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81 === 2025-08-21 === * 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:31 godog: reboot nfs workers to reset processes stuck in D state * 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 * 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 === 2025-08-20 === * 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) === 2025-08-19 === * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]]) * 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]]) * 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud === 2025-08-16 === * 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111 * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111 * 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111 * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111 === 2025-08-15 === * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 * 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 === 2025-08-14 === * 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) * 11:18 taavi: delete tools-prometheus-6, shutdown for a while * 08:51 godog: bounce stashbot * 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]] * 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]]) * 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]]) * 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]]) * 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 === 2025-08-08 === * 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 * 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67 * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-05 === * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 * 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 === 2025-08-01 === * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-07-30 === * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 15:56 andrewbogott: draining and restarting tools-k8s-worker-nfs-74 * 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrewbogott: draining and restarting tools-k8s-worker-nfs-58 and tools-k8s-worker-nfs-32 * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase * 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97) * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-07-28 === * 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]] === 2025-07-26 === * 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-23 === * 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-21 === * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-07-19 === * 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-07-18 === * 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 === 2025-07-14 === * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78 === 2025-07-13 === * 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 * 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 === 2025-07-11 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 * 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 === 2025-07-09 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]]) * 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]]) === 2025-07-08 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-07 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest * 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-06 === * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 * 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 === 2025-07-05 === * 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs === 2025-07-04 === * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-07-03 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-02 === * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55 * 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-06-30 === * 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) === 2025-06-28 === * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:08 dcaro: left a tmux running with a script to restart nginx if stuck * 09:59 dcaro: restarted nginx in tools-static === 2025-06-27 === * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 === 2025-06-26 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 * 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 === 2025-06-24 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-23 === * 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]] === 2025-06-22 === * 00:09 andrewbogott: rebooting tools-prometheus-8 === 2025-06-21 === * 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of D state * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-19 === * 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools === 2025-06-18 === * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable === 2025-06-16 === * 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 === 2025-06-14 === * 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-06-12 === * 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh) * 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-11 === * 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 === 2025-06-10 === * 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved === 2025-06-06 === * 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark * 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-06-05 === * 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert * 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-30 === * 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 07:38 taavi: reboot tools-static-15 to unstuck NFS things === 2025-05-24 === * 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 * 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 === 2025-05-23 === * 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 * 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 === 2025-05-22 === * 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:15 dcaro: restart tools-static nginx due to nfs hiccup * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7 * 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 * 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]] === 2025-05-21 === * 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh) * 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead === 2025-05-20 === * 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97) * 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown * 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-19 === * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-14 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 * 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-05-12 === * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512 ([[phab:T393686|T393686]]) * 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 * 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 === 2025-05-10 === * 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo<nowiki>{</nowiki>,.socket<nowiki>}</nowiki> # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]]) * 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-<nowiki>{</nowiki>pam,sudo<nowiki>}</nowiki>.service && systemctl restart sssd-pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket # try to reset the rate limits this way ([[phab:T393732|T393732]]) * 16:22 lucaswerkmeister: systemctl restart sssd-<nowiki>{</nowiki>pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>,sudo<nowiki>}</nowiki>.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) * 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam<nowiki>{</nowiki>,<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket<nowiki>}</nowiki> # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]? === 2025-05-09 === * 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake * 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory) * 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs === 2025-05-08 === * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state * 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-07 === * 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2 * 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2 * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:33 dcaro: released jobs-cli 16.1.12 * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 === 2025-05-04 === * 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06) === 2025-05-02 === * 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]] * 13:55 taavi: reboot tools-static-15 === 2025-04-28 === * 13:07 dhinus: tools-db-4: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:06 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:05 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T318479|T318479]] === 2025-04-24 === * 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12 * 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing "sudo: PAM account management error: Authentication service cannot retrieve authentication info" * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:51 taavi: add missing ICMPv6 security group rule to 'default' group * 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]] === 2025-04-23 === * 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles === 2025-04-21 === * 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]] === 2025-04-17 === * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-11 === * 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-10 === * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2025-04-09 === * 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff) * 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-04-07 === * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-06 === * 02:12 andrewbogott: truncating large logfiles on tools nfs === 2025-04-04 === * 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes === 2025-04-03 === * 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 === 2025-04-02 === * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 === 2025-04-01 === * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 === 2025-03-31 === * 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 * 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 === 2025-03-28 === * 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]]) * 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]]) * 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]]) * 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]]) === 2025-03-27 === * 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster * 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 === 2025-03-25 === * 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 * 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2 * 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 * 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-03-22 === * 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-18 === * 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) * 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) === 2025-03-17 === * 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) === 2025-03-16 === * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 === 2025-03-15 === * 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]] === 2025-03-13 === * 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 === 2025-03-12 === * 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o * 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-03-07 === * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-03-06 === * 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-03-05 === * 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values) * 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes * 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-03-04 === * 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual) * 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-03 === * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-01 === * 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 === 2025-02-27 === * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-02-26 === * 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-25 === * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 === 2025-02-24 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-21 === * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-02-20 === * 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-19 === * 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-02-18 === * 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]]) * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]]) === 2025-02-17 === * 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-10 === * 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-02-09 === * 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes === 2025-02-07 === * 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]] * 17:58 andrewbogott: "SET GLOBAL read_only=OFF; " on tools-db-4; both -5 and -4 were set to read only. No idea why or how... * 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 === 2025-02-06 === * 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage * 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-03 === * 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 === 2025-02-01 === * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102 * 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]] * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102 * 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]] * 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 * 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 === 2025-01-31 === * 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]] === 2025-01-29 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-27 === * 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-26 === * 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster * 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster * 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 * 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 === 2025-01-24 === * 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 === 2025-01-23 === * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs === 2025-01-22 === * 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2025-01-18 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2025-01-17 === * 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing) === 2025-01-15 === * 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-13 === * 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:49 dcaro: restart prometheus to pick up the new ips for vms and such * 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 * 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 * 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots === 2025-01-12 === * 09:53 taavi: hard reboot tools-k8s-worker-nfs-55 === 2025-01-08 === * 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22 * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22 * 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70 * 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 * 14:16 dcaro: reboot tools-static-15 nfs is stuck === 2025-01-07 === * 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor === 2025-01-06 === * 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-01-05 === * 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13 === 2025-01-03 === * 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 * 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69 * 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69 === 2025-01-02 === * 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 === 2025-01-01 === * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 === 2024-12-13 === * 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 * 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 === 2024-12-12 === * 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2024-12-06 === * 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]]) * 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]]) * 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing * 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main === 2024-11-29 === * 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-27 === * 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns # update resolv.conf in coredns containers === 2024-11-26 === * 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:22 dcaro: rebooting k8s-control-9 * 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:17 dcaro: rebooting k8s-control-8 * 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:14 dcaro: restarting tools-k8s-worker-nfs-72 * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 09:12 dcaro: restarting tools-k8s-worker-nfs-70 * 09:11 dcaro: restarting tools-k8s-worker-nfs-50 * 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 * 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]]) * 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]]) === 2024-11-25 === * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2024-11-23 === * 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]]) * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]]) * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-18 === * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-15 === * 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]]) * 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) === 2024-11-14 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-10 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]]) === 2024-11-06 === * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 dcaro: added api.svc.toolforge.org dns record entry * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-22 === * 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 * 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]] === 2024-10-16 === * 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-14 === * 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]]) * 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 * 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-10-09 === * 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup * 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating * 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]] * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-04 === * 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-02 === * 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-01 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:28 dcaro: updated ci image with latest precommit versions * 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-30 === * 18:25 taavi: run striker migrations [[phab:T359428|T359428]] === 2024-09-28 === * 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2024-09-24 === * 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-20 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]]) * 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network * 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-17 === * 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64 * 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63 * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62 * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60 * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6 * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56 * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49 * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31 * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30 * 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29 * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28 * 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster * 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]]) === 2024-09-16 === * 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-09-13 === * 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) === 2024-09-12 === * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) * 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) === 2024-09-11 === * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-09 === * 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager === 2024-09-06 === * 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6 * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-09-05 === * 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-03 === * 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 * 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 === 2024-09-02 === * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15 * 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15 * 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2 * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster * 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-08-26 === * 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster * 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster * 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 11:06 dcaro: manually deleted the coredns pods that had been around for 4d * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:18 dcaro: scale up cordens deployment to 4 replicas === 2024-08-21 === * 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-19 === * 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 === 2024-08-15 === * 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 * 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 === 2024-08-13 === * 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-08-12 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-06 === * 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2024-08-05 === * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot * 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected. * 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55 * 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc * 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]] * 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 * 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 === 2024-07-30 === * 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2024-07-29 === * 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) * 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance * 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission * 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 17:00 dcaro: moving the toolforge apt repo to tools-services-06 * 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud * 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-19 === * 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6 * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-07-18 === * 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16 * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16 * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-11 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1 * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0 * 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]] * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]]) * 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]]) * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23 * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23 * 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1 * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24 * 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24 * 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22 * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22 * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]]) * 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]] * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 === 2024-06-25 === * 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-> dict[str, Any]` type annotations causing [[phab:T368463|T368463]] * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104 * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103 * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56 * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55 * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54 * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53 * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52 * 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51 * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53 * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52 * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7 * 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7 * 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8' * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8' * 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9 * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9 * 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47 * 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44 * 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44 * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42 * 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41 * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39 * 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38 * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37 * 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36 * 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35 * 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35 * 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34 * 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33 * 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35 * 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33 * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33 * 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33 === 2024-06-24 === * 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots. * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32 * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29 * 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29 * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28 * 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28 * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27 * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27 * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10 * 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10 * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13 * 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13 * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26 * 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22 * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22 * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21 * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21 * 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19 * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20 * 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17 * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14 * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13 * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13 * 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12 * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12 * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7 * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8 * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8 * 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8 * 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15 * 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4 * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4 * 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10 * 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10 * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9 * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7 * 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7 * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5 * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7 * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7 * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7 * 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43 * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7 * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7 * 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7 * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43 * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7 * 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6 * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6 * 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1 * 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5 * 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5 * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1 * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6 * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2 * 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2 * 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2 * 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2 * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2 * 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05 * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8 * 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8 * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5 * 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5 * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-06-20 === * 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]] * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-14 === * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]]. * 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 13:45 taavi: hard reboot tools-k8s-control-7 * 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:57 dcaro: cleaning old maintain-kubeusers configmaps * 10:45 dcaro: cleaning up old resourcequotas === 2024-06-10 === * 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno === 2024-06-07 === * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 dcaro: deploying toolforge-webservice 0.103.6 * 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13 * 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12 === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-05-29 === * 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api * 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-28 === * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-25 === * 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-22 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-15 === * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO * 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9 * 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 === 2024-05-07 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-05 === * 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]] === 2024-04-24 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-17 === * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13 * 10:48 dcaro: rebooting tools-k8s-worker-nfs-1 === 2024-04-16 === * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0' * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0' === 2024-04-15 === * 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-11 === * 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-09 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission * 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6 === 2024-04-08 === * 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 * 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5 === 2024-04-05 === * 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]] === 2024-04-02 === * 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]]) * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05 === 2024-03-28 === * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05 * 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]] * 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06 * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06 * 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' === 2024-03-27 === * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01 === 2024-03-26 === * 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion' * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion' * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11 * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11 * 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]] === 2024-03-25 === * 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector * 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud === 2024-03-22 === * 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error) === 2024-03-21 === * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4 * 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3 * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04 * 12:30 taavi: move checker service address to tools-checker-5 * 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker' * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' === 2024-03-19 === * 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images) * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 * 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]] * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]]) === 2024-03-18 === * 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 taavi: restart harbor services after docker service restart * 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1 * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1 * 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]] * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]] === 2024-03-15 === * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48' * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48' * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01 * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]] * 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]] === 2024-03-13 === * 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable. === 2024-03-12 === * 12:38 taavi: hard reboot tools-prometheus-6 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-11 === * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]] === 2024-03-09 === * 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs === 2024-03-08 === * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-07 === * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-06 === * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates * 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs === 2024-03-05 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]]) * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud === 2024-03-04 === * 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state === 2024-03-03 === * 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state) === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-29 === * 14:36 dcaro: deploy webservice 0.103.3 === 2024-02-28 === * 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]]) * 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) === 2024-02-23 === * 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2 * 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]] === 2024-02-22 === * 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster * 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 === 2024-02-21 === * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4 * 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster === 2024-02-20 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster * 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 * 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 * 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster * 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100 * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster * 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96 * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95 * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95 * 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92 * 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster * 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91 * 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91 * 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90 * 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88 === 2024-02-19 === * 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster * 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84 * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83 * 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81 * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81 * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-16 === * 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80 * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77 * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77 === 2024-02-15 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 * 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 * 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 * 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster === 2024-02-14 === * 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30 * 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3 * 16:30 taavi: reboot tools-sgeexec-10-23 due to high load * 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74 * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster * 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71 * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71 * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster * 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70 * 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70 * 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69 * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68 * 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68 === 2024-02-13 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster * 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65 * 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64 * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64 === 2024-02-12 === * 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster * 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster * 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61 * 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60 * 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60 * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster * 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58 * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58 * 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54 * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15 * 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-11 === * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-02-09 === * 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule * 13:06 taavi: reboot tools-sgecron-2 due to high load * 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config * 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 * 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes === 2024-02-08 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 * 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42 === 2024-02-07 === * 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers * 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers === 2024-02-06 === * 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-01-31 === * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-30 === * 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster * 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster * 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster * 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41 * 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41 * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40 * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40 * 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39 * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39 * 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38 * 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38 * 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37 * 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37 * 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]]) * 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]]) * 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]] * 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]]) * 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]]) * 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]]) === 2024-01-29 === * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud * 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud * 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster * 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33 * 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33 * 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32 * 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32 * 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31 * 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30 * 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster * 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB * 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB * 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-26 === * 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta === 2024-01-25 === * 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1 === 2024-01-23 === * 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22 * 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-18 === * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17 === 2024-01-17 === * 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]] * 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]]) * 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]]) * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]] === 2024-01-15 === * 09:18 taavi: reboot stuck tools-k8s-worker-84 === 2024-01-12 === * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12' * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12' === 2024-01-11 === * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-10 === * 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:17 taavi: reboot tools-k8s-worker-98 === 2024-01-09 === * 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor -- [[phab:T354714|T354714]] * 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it) * 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder * 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder * 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:13 taavi: reboot tools-sgeexec-10-17 due to high load === 2024-01-08 === * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28 * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 taavi: reboot tools-sgeexec-10-21 === 2024-01-05 === * 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-04 === * 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3 === 2024-01-03 === * 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs * 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage === 2024-01-02 === * 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]]) * 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]]) * 10:37 dcaro: hard reboot tools-harbor-1 * 10:13 dhinus: hard reboot tools-harbor-1 === 2024-01-01 === * 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]] ==Archives== * [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014) * [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017) * [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019) * [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021) * [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023) </noinclude> {{SAL|Project Name=tools}} <noinclude>[[Category:SAL]]</noinclude> qm9zcsr5lnx5bff01z9uyoiq6do3j29 2452779 2452778 2026-08-30T20:56:38Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2452779 wikitext text/x-wiki === 2026-08-30 === * 20:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:26 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:43 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:17 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:15 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:03 dhinus: restart harbor in tools-harbor-2 with "docker compose restart" === 2026-08-26 === * 17:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 17:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 17:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 17:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 17:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T433132|T433132]]) * 16:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools- * 16:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T433132|T433132]]) * 16:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for tools-k8s-control-7 ([[phab:T433132|T433132]]) * 15:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.32.13 to 1.33.13 ([[phab:T433132|T433132]]) * 15:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.33.13 ([[phab:T433132|T433132]]) * 10:00 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:17 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-24 === * 10:46 taavi: disable access to elastic cluster as announced on cloud-announce [[phab:T401818|T401818]] === 2026-08-20 === * 16:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:41 dcaro: tools-puppetserver-01 was out of memory because of java and killed sssd backend * 15:33 dcaro: restart tools-puppetserver-01, unresponsive * 14:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:38 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:29 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:48 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 13:42 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:03 dcaro: truncate haproxy.log to 1G (out of space) on haproxy-8 === 2026-08-14 === * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-gateway-3 * 10:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-gateway-3 * 10:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8, tools-k8s-gateway-2 * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8, tools-k8s-gateway-2 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7, tools-k8s-gateway-1 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7, tools-k8s-gateway-1 === 2026-08-13 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 11:25 taavi: update striker to 2026-08-13-111038-production * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 09:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2026-08-12 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 13:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-05 === * 14:43 taavi: update striker to 2026-08-04-183239-production === 2026-08-04 === * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry (exit_code=0) for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 16:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_certmanager_images_to_registry for certmanager 1.20.3 ([[phab:T408786|T408786]]) * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:22 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:15 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-03 === * 20:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, t * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-48, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 17:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-66 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T433426|T433426]]) * 17:06 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers ([[phab:T433426|T433426]]) * 15:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T433426|T433426]]) * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-31 === * 13:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:03 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:36 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:28 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:18 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 15:10 godog: hard reboot tools-bastion-15 * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:05 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:04 dcaro: reboot tools-bastion-15, getting out of memory * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:00 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:29 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 09:24 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 09:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 09:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 09:16 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 * 09:16 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10,tools-k8s-worker-nfs-36 === 2026-07-29 === * 09:54 godog: test dumps-nfs failover - [[phab:T432325|T432325]] * 09:48 godog: enable 'dumps_use_nfs_lb: true' for nfs worker prefix - [[phab:T432325|T432325]] * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T433184|T433184]]) * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T433184|T433184]]) * 08:56 godog: enable 'dumps_use_nfs_lb: true' for bastion/static/mail prefixes - [[phab:T432325|T432325]] === 2026-07-28 === * 08:25 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, t * 06:56 godog: resume nfs workers roll restart - [[phab:T432325|T432325]] * 06:55 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs === 2026-07-27 === * 17:18 dcaro: increase object storage object number quota to 1M ([[phab:T433281|T433281]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:06 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 11:39 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 11:37 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) * 08:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) === 2026-07-23 === * 17:11 dcaro: resize haproxy-8 * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:55 dcaro: resize haproxy-7 * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 32 cores, 65536 ram * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 32 cores, 65536 ram * 12:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-07-20 === * 12:28 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-3.tools.eqiad1.wikimedia.cloud * 12:26 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-2.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 12:22 taavi@cloudcumin2001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-opensearch-1.tools.eqiad1.wikimedia.cloud * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:43 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:33 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-16 === * 15:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-5 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-5 * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-10.tools.eqiad1.wikimedia.cloud * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 11:38 taavi: failing over redis to new trixie cluster [[phab:T360704|T360704]] * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-6 * 11:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-6 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-redis-7 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-redis-7 * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-cumin-1 * 11:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-cumin-1 * 01:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 19:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 19:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-cumin-2.tools.eqiad1.wikimedia.cloud * 18:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-cumin' * 18:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-cumin' * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-9.tools.eqiad1.wikimedia.cloud * 14:40 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:26 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-redis-8.tools.eqiad1.wikimedia.cloud * 13:49 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-redis-8.tools.eqiad1.wikimedia.cloud * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:00 dcaro: restart haproxy on haproxy-8 removing tune.disable-zero-copy-forwarding from config === 2026-07-12 === * 21:01 dcaro: restart haproxy on haproxy-8 * 06:28 dcaro: restart haproxy on haproxy-8 === 2026-07-11 === * 17:34 dcaro: rebooting haproxy-8 === 2026-07-09 === * 17:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 12:53 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:27 taavi: rebooting bastions for kernel upgrades * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 10:18 taavi: deploy latest striker with [[phab:T430120|T430120]] [[phab:T409341|T409341]] fixes * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 17:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 08:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) === 2026-06-24 === * 16:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-15 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 12:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T427919|T427919]]) * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:45 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T427919|T427919]]) * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.31.14 to 1.32.13 ([[phab:T427919|T427919]]) * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 11:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.32.13 ([[phab:T427919|T427919]]) * 09:53 godog: delete tools-checker puppet prefix - [[phab:T313030|T313030]] * 09:52 godog: delete tools-checker-5 - [[phab:T313030|T313030]] === 2026-06-11 === * 13:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 13:18 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-09 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 11:01 taavi: starting Kyverno upgrade [[phab:T427928|T427928]] === 2026-06-04 === * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:06 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kubectl:v1.32.7 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.16.4 * 12:05 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.16.4 * 12:04 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:03 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:02 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.16.4 * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.16.4 * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry === 2026-06-02 === * 15:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 10:33 taavi: migrate acme-chief, puppetdb to nftables [[phab:T427799|T427799]] * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 20:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 20:33 andrewbogott: rebuilt pywikibot for v11.3.0 [[phab:T426547|T426547]] * 12:12 godog: begin roll-restart webservices with new default requests - [[phab:T420565|T420565]] * 12:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:02 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 09:11 godog: delete obsolete limitranges - [[phab:T427352|T427352]] === 2026-05-21 === * 12:46 taavi: renew prometheus certificate * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-gateway-1, tools-k8s-gateway-2, tools-k8s-gateway-3 * 10:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:48 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 00:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-05-20 === * 23:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 10:16 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-16 === * 00:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers === 2026-05-15 === * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 19:02 taavi: rebooting bastions and k8s workers to pick up kernel updates === 2026-05-14 === * 16:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-13 === * 12:07 godog: resume restarting webservices using default memory requests - [[phab:T420565|T420565]] * 08:46 godog: restart sample webservices with new memory requests https://phabricator.wikimedia.org/P92497 - [[phab:T420565|T420565]] * 08:36 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 22:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2026-05-11 === * 00:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 00:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-07 === * 12:02 taavi: draining tools-k8s-worker-106 to investigate [[phab:T425172|T425172]] === 2026-05-05 === * 04:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 02:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 10:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 15:59 andrewbogott: hard rebooting tools-puppetserver-01.tools, it seems to have crashed * 09:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:12 taavi: uninstall ingress-nginx-gen2 from the cluster [[phab:T392356|T392356]] * 08:08 taavi: delete all ingress objects [[phab:T392356|T392356]] === 2026-04-21 === * 14:06 taavi: save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json [[phab:T392356|T392356]] * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-16 === * 13:09 taavi: bump istio traffic percentage 75% -> 100% [[phab:T392356|T392356]] === 2026-04-15 === * 10:45 taavi: bump istio traffic percentage 50% -> 75% [[phab:T392356|T392356]] === 2026-04-13 === * 09:11 taavi: bump istio traffic percentage 25% -> 50% [[phab:T392356|T392356]] * 07:33 taavi: bump istio traffic percentage 10% -> 25% [[phab:T392356|T392356]] === 2026-04-10 === * 14:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api === 2026-04-07 === * 23:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 19:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 19:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:18 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:59 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:57 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 15:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T361237|T361237]]) * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrewbogott: replacing etcd nodes with bookworm-based VMs * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 18:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-30 === * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 dcaro: removing wal from prometheus nodes to restart them === 2026-03-26 === * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 17:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/cadvisor:0.56.2 * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-25 === * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-package-builder-04 * 13:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-package-builder-04 === 2026-03-24 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 11:16 taavi: send 10% of traffic to istio [[phab:T392356|T392356]] * 10:53 taavi: send 5% of traffic to istio [[phab:T392356|T392356]] * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 20:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:07 taavi: fixing 2 tools still running ruby2.1 image to use that instead of 'ruby2' in service.manifest * 08:52 taavi: fixing 2 tools still running ruby2.5 image to use that instead of 'ruby25' in service.manifest * 08:49 taavi: fixing 12 tools still running node6 image to use that instead of 'nodejs' in service.manifest * 08:38 taavi: fixing 12 tools still running golang1.11 image to use that instead of 'golang111' in service.manifest * 08:36 taavi: fixing 60 tools still running python3.4 image to use 'python3.4' instead of 'python' in service.manifest === 2026-03-18 === * 12:00 taavi: restarting existing web services to backfill HTTPRoute resources [[phab:T392356|T392356]] * 07:37 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:37 filippo@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-83.tools.eqiad1.wikimedia.cloud to the cluster * 07:23 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T419824|T419824]]) === 2026-03-17 === * 12:43 taavi: shutdown tools-package-builder-04 [[phab:T401819|T401819]] === 2026-03-15 === * 03:10 andrewbogott: rebooting tools-redis-6, VM is in state ERROR === 2026-03-13 === * 22:04 taavi: reboot tools-bastion-15 [[phab:T420044|T420044]] * 19:06 taavi: reboot tools-bastion-14 [[phab:T420044|T420044]] === 2026-03-12 === * 13:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 13:50 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2026-03-10 === * 11:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:30 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-3.tools.eqiad1.wikimedia.cloud to the cluster * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:19 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-2.tools.eqiad1.wikimedia.cloud to the cluster * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the tools cluster * 13:07 taavi@cloudcumin1001: Added a new k8s gateway tools-k8s-gateway-1.tools.eqiad1.wikimedia.cloud to the cluster * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the tools cluster === 2026-03-06 === * 11:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-03-05 === * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2026-03-04 === * 20:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:44 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:29 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:14 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:13 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:12 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=97) * 19:11 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:08 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 19:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 18:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 18:17 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:38 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 14:54 dcaro: increase object quota to 400k ([[phab:T418528|T418528]]) * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2026-03-03 === * 20:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 20:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component gateway-api * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-25 === * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.29.0 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 * 14:09 taavi: taavi@tools-imagebuilder-2:~$ sudo docker system prune -a # reclaiming disk space * 14:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=99) for Istio 1.29.0 * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.29.0 === 2026-02-24 === * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-20 === * 20:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 20:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-runner:24 * 19:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image toolsbeta-harbor.wmcloud.org/toolforge/heroku-builder:22_{{Gerrit|20240105}} === 2026-02-11 === * 21:49 taavi: remove hiera override still allowing ssh agent forwarding onto toolforge bastions [[phab:T198138|T198138]] === 2026-02-05 === * 19:08 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:48 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:42 volans: re-enabling puppet on NFS workers to update the infra-tracing-nfs === 2026-02-04 === * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 14:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 * 14:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.14.3 === 2026-02-03 === * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry for image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.7 === 2026-01-28 === * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-14 === * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-15.tools.eqiad1.wikimedia.cloud, tools-bastion-14.tools.eqiad1.wikimedia.cloud ([[phab:T413797|T413797]]) * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 15:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 15:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 15:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T413797|T413797]]) * 14:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:43 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112, tools-k8s-worker-113 ([[phab:T413797|T413797]]) * 14:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.30.14 to 1.31.14 ([[phab:T413797|T413797]]) * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) * 13:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade to 1.31.14 ([[phab:T413797|T413797]]) === 2026-01-12 === * 17:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 15:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 14:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 13:48 andrewbogott: removing tools-k8s-etcd-24 in prep for rebuilding cloudvirtlocal1003 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 03:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:53 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:59 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 01:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:01 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 22:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 200k * 11:05 godog: bump object quota to 500G === 2025-12-17 === * 17:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.12.1 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.6.3 ([[phab:T399313|T399313]]) * 17:53 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.6.3, Alloy 1.12.1 ([[phab:T399313|T399313]]) === 2025-12-15 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.17.0 ([[phab:T412695|T412695]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T412695|T412695]]) === 2025-12-14 === * 02:14 andrewbogott: running 'kubectl rollout restart -n envvars-admission deployment/envvars-admission' in response to an envvars alert === 2025-12-11 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 21:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 21:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 20:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 20:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 20:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 19:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 19:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) === 2025-12-03 === * 19:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) === 2025-12-02 === * 20:22 andrewbogott: stop/starting harbordb1 to fix presumed mtu mismatch * 20:06 andrewbogott: rebooting tools-harbordb1 to aid with host draining * 08:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.11.3 ([[phab:T399313|T399313]]) * 08:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.11.3 ([[phab:T399313|T399313]]) === 2025-12-01 === * 22:31 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.4.0 ([[phab:T399313|T399313]]) * 22:30 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Alloy 1.4.0 ([[phab:T399313|T399313]]) * 16:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:26 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-27 === * 12:15 volans: [continue] on the haproxy nodes * 12:15 volans: temporarily disabling puppet to deploy gerrit {{Gerrit|1211610}} === 2025-11-26 === * 14:48 volans: enabled infra-tracing-nfs on all nfs workers after testing it on few hosts * 09:46 dhinus: restarting tools-db-6 to apply a config change [[phab:T409922|T409922]] === 2025-11-25 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 02:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:24 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 10:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 18:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 * 17:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:55 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:45 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:36 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:56 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:51 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:37 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:37 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 15:17 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:01 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 14:46 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 14:40 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-19 === * 16:19 andrewbogott: increased object count quota to 100,000 * 16:03 andrewbogott: increased object storage quota to 200GB * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-11-18 === * 18:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-harbor-2 (cluster eqiad1) * 14:36 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-harbor-2 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-9 (cluster eqiad1) * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-9 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-113 (cluster eqiad1) * 14:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-113 (cluster eqiad1) * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-113 * 14:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-112 (cluster eqiad1) * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-112 (cluster eqiad1) * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-112 * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-82 (cluster eqiad1) * 14:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-82 * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-81 (cluster eqiad1) * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-81 * 14:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-81 * 14:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-worker-nfs-80 (cluster eqiad1) * 14:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-80 * 14:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-80 * 14:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-prometheus-8 (cluster eqiad1) * 14:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-prometheus-8 (cluster eqiad1) * 12:05 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:56 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:24 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:19 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legacy-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.instance.stop_start (exit_code=97) vm tools-legaci-redirector-3 (cluster eqiad1) * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-legaci-redirector-3 (cluster eqiad1) === 2025-11-17 === * 18:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T409981|T409981]]) * 10:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T409981|T409981]]) === 2025-11-14 === * 16:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-4 ([[phab:T409287|T409287]]) * 16:26 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-4 ([[phab:T409287|T409287]]) === 2025-11-13 === * 15:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.instance.stop_start (exit_code=99) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-bastion-14 (cluster eqiad1) * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-bastion-14 (cluster eqiad1) * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-7 (cluster eqiad1) * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm tools-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm tools-k8s-haproxy-8 (cluster eqiad1) === 2025-11-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-11 === * 15:28 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.7 ([[phab:T399313|T399313]]) * 15:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.7 ([[phab:T399313|T399313]]) === 2025-11-10 === * 22:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 22:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:16 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:14 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest * 22:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) * 22:08 raymond-ndibe@cloudcumin1001: Updating container image toolsbeta-harbor.wmcloud.org/toolforge-pre-built/toolforge-bookworm-sssd:latest:latest * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-07 === * 11:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:35 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) * 11:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-7 ([[phab:T409287|T409287]]) * 11:33 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-7 ([[phab:T409287|T409287]]) === 2025-11-06 === * 16:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 taavi: taavi@tools-bastion-15:~ $ sudo loginctl terminate-user damian * 14:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:53 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-7.tools.eqiad1.wikimedia.cloud ([[phab:T409287|T409287]]) * 14:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T409287|T409287]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T409287|T409287]]) === 2025-11-04 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 17:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-10-30 === * 18:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:39 taavi: kick off script to rebuild all pre-built images, including [[phab:T407707|T407707]] * 16:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 16:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 16:27 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T408669|T408669]]) * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry (exit_code=0) for Calico v3.29.6 * 12:48 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/typha:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/node:v3.29.6 * 12:47 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/kube-controllers:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/ctl:v3.29.6 * 12:46 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/calico/cni:v3.29.6 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.calico.copy_images_to_registry for Calico v3.29.6 * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:07 taavi: delete paws, paws-master security groups, long obsolete as paws is now in a separae project * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 14:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-10-27 === * 22:09 taavi: copy toolviews database hiera data to a place where haproxy nodes can see them [[phab:T408454|T408454]] * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:16 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-24 === * 20:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, t * 18:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-35, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-41, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-44, tools-k8s-worker-nfs * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-27 * 17:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k * 16:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-9, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-16, to * 16:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105, tools-k8s-worker-106 * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105, tools-k8s-worker-106 * 16:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-102,tools-k8s-worker-103 * 16:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102,tools-k8s-worker-103 * 13:37 andrewbogott: rebooting clouddumps100[12] for [[phab:T407110|T407110]] === 2025-10-23 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:11 taavi: deleting old nginx front proxy instances [[phab:T283948|T283948]] * 10:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 15:56 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/nginx-ingress-controller:v1.13.3 * 15:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:35 taavi: moving toolforge traffic to haproxy directly [[phab:T283948|T283948]] * 07:00 godog: delete tools-nfs-2 - [[phab:T404584|T404584]] === 2025-10-21 === * 08:53 godog: shut down tools-nfs-2 - [[phab:T404584|T404584]] * 07:52 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 07:49 godog: resize tools-nfs-3 to match tools-nfs-2 (g4.cores16.ram64.disk20.10xiops) - [[phab:T404584|T404584]] * 00:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-20 === * 16:31 taavi: make logrotate run hourly on haproxy nodes [[phab:T284558|T284558]] === 2025-10-16 === * 12:01 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-15 === * 08:03 godog: tools-nfs-3 is back - [[phab:T404584|T404584]] * 08:00 godog: resize tools-nfs-3 - [[phab:T404584|T404584]] === 2025-10-14 === * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:45 godog: update nfs-tools.wmcloud.org and nfs.svc.toolforge.org proxied to point to tools-nfs-3 === 2025-10-13 === * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:17 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-72, tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-8, tools-k8s-worker-nfs-80, tools-k8s-worker-nfs-81, tools-k8s-worker-nfs-82, too * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:14 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-81 (cluster eqiad1, project tools) * 09:13 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:13 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-69 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-68 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-67 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:08 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-66 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-65 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:07 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-61 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-58 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-57 (cluster eqiad1, project tools) * 09:06 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-55 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-54 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-53 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-50 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-5 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-48 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-47 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:02 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-46 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-45 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-44 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-43 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:01 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-42 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-41 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-40 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 09:00 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-39 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:59 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-38 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:58 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:57 filippo@cloudcumin1001: START - Cookbook wmcs.vps.instance.force_reboot vm tools-k8s-worker-nfs-37 (cluster eqiad1, project tools) * 08:10 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:05 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) ([[phab:T404584|T404584]]) * 08:03 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service ([[phab:T404584|T404584]]) * 08:01 godog: switch NFS from tools-nfs-2 to tools-nfs-3 - [[phab:T404584|T404584]] * 07:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-76, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-81 === 2025-10-10 === * 09:22 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 * 09:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-1 === 2025-10-09 === * 08:21 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 08:15 filippo@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2025-10-08 === * 21:47 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 21:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-76 * 12:27 godog: very brief nfs interruption to wrap up [[phab:T347681|T347681]] * 10:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 06:55 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 * 06:43 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-71 === 2025-10-07 === * 18:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-11 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-11 * 15:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-69 * 15:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-69 * 14:51 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 14:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-65, tools-k8s-worker-nfs-69 * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:59 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:58 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-06 === * 12:06 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-7 * 11:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-7 * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 08:19 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:18 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 * 07:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-36, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-76 === 2025-10-03 === * 12:51 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-6 * 12:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-6 * 12:45 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-5 * 09:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-5 === 2025-10-02 === * 13:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-haproxy-7 * 13:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 09:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 * 08:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-42, tools-k8s-worker-nfs-55 === 2025-10-01 === * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-30 === * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 08:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:19 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 * 08:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-67 === 2025-09-29 === * 13:23 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 13:23 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 11:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 10:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-79 * 07:00 godog: kick stuck nfs workers from clouddumps1001 === 2025-09-28 === * 08:54 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-23, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-7, tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:13 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T405850|T405850]]) * 08:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-1 ([[phab:T405850|T405850]]) * 08:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:08 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:08 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:07 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 08:07 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-25 === * 18:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 20:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 20:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:30 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-nfs-43 * 17:07 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-nfs-43 * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 16:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 ([[phab:T400957|T400957]]) * 13:49 dcaro: patched all tools with new resource defaults, everything looks good * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro: depolyed jobs-api change to default resources, patching existing jobs * 13:08 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 03:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 03:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 === 2025-09-23 === * 20:08 andrewbogott: creating puppetdbpostgres and adding it to tools-puppetdb-2 to store postgres data; the root volume of that VM was filling up and causing widespread puppet issues * 01:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-09-22 === * 16:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-21 === * 09:17 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-2 * 03:16 dcaro: acking and silencing CPU capacity alerts to handle on Monday, they should not page * 01:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 01:46 andrew@cloudcumin1001: Added a new k8s worker tools-k8s-worker-113.tools.eqiad1.wikimedia.cloud to the cluster * 01:36 andrewbogott: adding additional worker node in response to repeated capacity alerts * 01:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-09-19 === * 13:09 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 13:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 === 2025-09-18 === * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 11:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 08:52 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 08:34 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-3 * 06:47 taavi: delete tools-sgebastion-10 [[phab:T314665|T314665]] === 2025-09-17 === * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 12:53 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-32 * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm tools-prometheus-9 (cluster eqiad1, project tools) * 09:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:34 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 08:23 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 * 08:08 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-66, tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-10 === 2025-09-16 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:57 taavi: delete tools-sgebastion puppet prefix [[phab:T314665|T314665]] * 15:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 taavi: shutdown tools-sgebastion-10 [[phab:T314665|T314665]] * 14:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-13 * 14:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-13 * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-bastion-12 * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-bastion-12 * 07:11 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-75 * 06:57 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 06:51 filippo@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-09-15 === * 16:22 taavi: reboot old bastions to kick long-living connections into newer ones * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:47 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 * 12:35 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-66 === 2025-09-12 === * 08:49 taavi: pointing login.toolforge.org to tools-bastion-15 [[phab:T392510|T392510]] * 08:33 taavi: pointing dev.toolforge.org to tools-bastion-14 [[phab:T392510|T392510]] * 07:14 godog: uncordon tools-k8s-worker-nfs-53 after failed cookbook (?) yesterday === 2025-09-11 === * 14:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:36 godog: drain/reboot tools-k8s-worker-nfs-46 - [[phab:T404322|T404322]] * 14:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:22 andrewbogott: actually I didn't drain tools-k8s-worker-nfs-53 because the alert cleared on its own * 14:21 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-53 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 14:21 andrewbogott: draining/rebooting tools-k8s-worker-nfs-53 because of procs in D state * 13:42 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 13:36 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-14.tools.eqiad1.wikimedia.cloud * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-15.tools.eqiad1.wikimedia.cloud * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-09-10 === * 14:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:28 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:45 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:31 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers ([[phab:T403964|T403964]]) * 12:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T403964|T403964]]) === 2025-09-09 === * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:55 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) ([[phab:T404047|T404047]]) * 08:55 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot ([[phab:T404047|T404047]]) === 2025-09-08 === * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:16 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for tools-bastion-12.tools.eqiad1.wikimedia.cloud, tools-bastion-13.tools.eqiad1.wikimedia.cloud ([[phab:T402378|T402378]]) * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 ([[phab:T402378|T402378]]) * 11:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 10:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 10:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68: ([[phab:T402378|T402378]]) * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wor * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:40 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=97) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-w * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-wo * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-10, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-13, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-23, tools-k * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for tools-k8s-worker-102, tools-k8s-worker-103, tools-k8s-worker-105, tools-k8s-worker-106, tools-k8s-worker-107, tools-k8s-worker-108, tools-k8s-worker-109, tools-k8s-worker-110, tools-k8s-worker-111, tools-k8s-worker-112 ([[phab:T402378|T402378]]) * 08:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.29.15 to 1.30.14 ([[phab:T402378|T402378]]) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests === 2025-09-06 === * 23:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 * 23:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 === 2025-09-05 === * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 19:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:52 dcaro: added 'disable-ssl' to tools replica.my.cnf === 2025-09-03 === * 17:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 09:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 17:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-29 === * 15:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance abogott-nstesting * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance abogott-nstesting === 2025-08-28 === * 16:52 taavi: rebuild tcl, mariadb images on top of trixie [[phab:T400256|T400256]] * 08:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:05 taavi: copy missing aptly packages to trixie-<nowiki>{</nowiki>tools,toolsbeta<nowiki>}</nowiki> * 11:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-26 === * 13:42 dcaro: extended object storage quota to 100G ([[phab:T402923|T402923]]) * 10:25 dhinus: shut down tools-harbor-1 (no longer used) === 2025-08-25 === * 22:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-81 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-81 === 2025-08-21 === * 12:28 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:31 godog: reboot nfs workers to reset processes stuck in D state * 07:28 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 04:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 * 03:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-80 === 2025-08-20 === * 08:09 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) === 2025-08-19 === * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 15:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 15:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:50 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:37 dcaro: flipped the tools-harbor.wmcloud.org endpoint to point to tools-harbor-2 ([[phab:T350687|T350687]]) * 14:22 Raymond_Ndibe: setting tools-harbor-1 as read-only ([[phab:T350687|T350687]]) * 13:24 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 13:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/kube-state-metrics:v2.16.0 ([[phab:T362869|T362869]]) * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/metrics-server:v0.7.2 ([[phab:T362869|T362869]]) * 17:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362869|T362869]]) * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud ([[phab:T350687|T350687]]) * 16:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:35 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:34 wmbot~dcaro@acme: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud === 2025-08-16 === * 21:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-111 * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-111 * 21:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-111 * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-111 === 2025-08-15 === * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 19:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 * 19:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-103 === 2025-08-14 === * 15:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=99) for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-107, tools-k8s-worker-nfs-11, tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-3, tools-k8s-worker-nfs-41 * 11:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 11:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found * 02:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:36 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) * 11:18 taavi: delete tools-prometheus-6, shutdown for a while * 08:51 godog: bounce stashbot * 08:33 godog: refresh machine-id on tools-k8s-worker-[102-103,105-112].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-nfs-[1-3,5,7-14,16-17,19,21-24,26-27,32-48,50,53-55 ,57-58,61,65-82].tools.eqiad1.wikimedia.cloud === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:34 taavi: building initial trixie based images [[phab:T400255|T400255]] * 12:50 dcaro: redepoly kyverno ([[phab:T394787|T394787]]) * 12:49 dcaro: manually migrate cleanuppolicies.kyverno.io and clustercleanuppolicies.kyverno.io (using kyverno cli) ([[phab:T394787|T394787]]) * 10:01 dcaro: starting upgrade for kyverno ([[phab:T394787|T394787]]) * 10:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for tools-k8s-worker-103, tools-k8s-worker-nfs-36 * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-harbor-2.tools.eqiad1.wikimedia.cloud * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-58 === 2025-08-08 === * 06:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 * 14:20 andrewbogott: draining and rebooting tools-k8s-worker-nfs-67 * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:54 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:41 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:39 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli ([[phab:T401014|T401014]]) * 17:38 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-05 === * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-82, tools-k8s-worker-nfs-24 * 09:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 03:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 02:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 02:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 02:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 01:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 11:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 * 11:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-79, tools-k8s-worker-nfs-2 === 2025-08-01 === * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 16:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 04:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 04:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-07-30 === * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 15:56 andrewbogott: draining and restarting tools-k8s-worker-nfs-74 * 15:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 15:44 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58, tools-k8s-worker-nfs-32 * 15:32 andrewbogott: draining and restarting tools-k8s-worker-nfs-58 and tools-k8s-worker-nfs-32 * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:16 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:16 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-82.tools.eqiad1.wikimedia.cloud to the cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 13:06 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:05 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-81.tools.eqiad1.wikimedia.cloud to the cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 12:40 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.quota_increase * 12:39 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:31 wmbot~dcaro@acme: Added a new k8s worker-nfs tools-k8s-worker-nfs-80.tools.eqiad1.wikimedia.cloud to the cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:22 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:00 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:54 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=97) * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:29 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:28 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:07 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:59 wmbot~dcaro@acme: Added a new k8s worker tools-k8s-worker-112.tools.eqiad1.wikimedia.cloud to the cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:49 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 08:15 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:14 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2025-07-28 === * 20:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:58 taavi: update pywikibot image to 10.2.0 [[phab:T396933|T396933]] === 2025-07-26 === * 07:16 wmbot~root@toolforge: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:16 wmbot~root@toolforge: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-23 === * 18:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-21 === * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-07-19 === * 13:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 13:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-07-18 === * 10:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-44 === 2025-07-14 === * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-78 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-78 === 2025-07-13 === * 03:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 * 03:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73, tools-k8s-worker-nfs-24, tools-k8s-worker-nfs-12 === 2025-07-11 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 * 09:25 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-77, tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-37 === 2025-07-09 === * 17:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:55 dcaro: adding arch arm64 to all toolforge repos ([[phab:T398016|T398016]]) * 09:40 dcaro: added arch arm64 to jessie-tools repo ([[phab:T398016|T398016]]) === 2025-07-08 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-07 === * 17:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-53 * 17:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-53 * 16:39 dcaro: pushed new ci image docker-registry.svc.toolforge.org/cloud-cicd-py3.11-bookworm-tox:latest * 16:05 dcaro: clearing images from tools-imagebuilder-2 as it's out of space * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-06 === * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 * 16:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75, tools-k8s-worker-nfs-8 === 2025-07-05 === * 00:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-57 * 00:31 andrewbogott: restarting tools-k8s-worker-nfs-55 tools-k8s-worker-nfs-47 tools-k8s-worker-nfs-57, too many D state procs === 2025-07-04 === * 14:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 14:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12, tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-07-03 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 13:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 08:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-02 === * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 13:30 andrewbogott: restarting stuck tools tools-k8s-worker-nfs-74 tools-k8s-worker-nfs-39 tools-k8s-worker-nfs-55 * 13:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-55 * 10:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 14:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:31 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:30 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-5 ([[phab:T398170|T398170]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-06-30 === * 23:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 22:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-14 * 13:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 13:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69, tools-k8s-worker-nfs-70 * 10:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:46 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398170|T398170]]) * 10:45 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398170|T398170]]) * 10:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T398170|T398170]]) * 10:43 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T398170|T398170]]) === 2025-06-28 === * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67 * 10:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 * 10:08 dcaro: left a tmux running with a script to restart nginx if stuck * 09:59 dcaro: restarted nginx in tools-static === 2025-06-27 === * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 === 2025-06-26 === * 16:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 18:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:52 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:50 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 02:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 * 02:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-38 === 2025-06-24 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 15:06 andrewbogott: rebooting tools-k8s-worker-nfs-33, stuck processes * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 15:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-23 === * 09:08 taavi: restrict logging in to tools-sgebastion-10 (aka login-buster) [[phab:T397459|T397459]] === 2025-06-22 === * 00:09 andrewbogott: rebooting tools-prometheus-8 === 2025-06-21 === * 16:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 15:58 andrewbogott: rebooting tools-k8s-worker-nfs-54 tools-k8s-worker-nfs-12, lots of D state * 15:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-12 * 10:09 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:27 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:27 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 09:26 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-19 === * 18:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 17:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:56 dcaro: reboot tools-sgebastion-10 as it's stuck on NFS for some tools === 2025-06-18 === * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 04:22 andrewbogott: rebooting tools-prometheus-8; unreachable === 2025-06-16 === * 17:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 12:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 12:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 === 2025-06-14 === * 16:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-06-12 === * 10:36 dcaro: rebooting tools-prometheus-8 due to the VM having load issues (not responding to ssh) * 10:34 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:28 wmbot~dcaro@acme: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2025-06-11 === * 13:39 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=0) for Loki 3.5.0, Alloy 1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/alloy:v1.9.1 * 11:18 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry (exit_code=99) for Loki 3.5.0, Alloy 1.9.1 * 11:09 taavi@cloudcumin1001: Updating container image docker-registry.svc.toolforge.org/grafana/loki:3.5.0 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.logging.copy_images_to_registry for Loki 3.5.0, Alloy 1.9.1 === 2025-06-10 === * 17:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 17:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:41 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:26 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 16:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:45 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:15 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:48 taavi: add AAAA records to tools/toolsbeta-harbor proxies, previous monitoring issues resolved === 2025-06-06 === * 21:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 21:40 andrewbogott: restarting tools-prometheus-9 and tools-prometheus-8, lots of tools metrics just went dark * 21:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-74 * 18:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-06-05 === * 22:24 andrewbogott: running /srv/tools/cleanup.sh on tools-nfs-2 in a screen session, trying to clear disk space alert * 15:06 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:53 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-30 === * 16:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-11 * 15:28 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 07:38 taavi: reboot tools-static-15 to unstuck NFS things === 2025-05-24 === * 12:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 * 12:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 === 2025-05-23 === * 16:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-65 * 03:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 * 02:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54, tools-k8s-worker-nfs-37, tools-k8s-worker-nfs-43 === 2025-05-22 === * 21:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-45, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-55 * 20:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 19:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2, tools-k8s-worker-nfs-53, tools-k8s-worker-nfs-47, tools-k8s-worker-nfs-78, tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-1, tools-k8s-worker-nfs-21 * 19:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:15 dcaro: restart tools-static nginx due to nfs hiccup * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-8 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-8 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-7 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-7 * 07:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=1) for instance toolsbeta-prometheus-1 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 * 07:33 taavi: add AAAA record on *.toolforge.org [[phab:T211575|T211575]] === 2025-05-21 === * 15:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-10.tools.eqiad1.wikimedia.cloud * 15:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-proxy-9.tools.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-9.tools.eqiad1.wikimedia.cloud * 09:27 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/busybox:1.35 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/bitnami-kubectl:1.30.2 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:26 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-background-controller:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:25 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 09:25 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:04 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 09:04 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 09:03 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 09:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 08:54 dcaro: deployed the new dns entry for docker-registry.svc.toolforge.org (might take some time to refresh) * 08:47 dcaro: deleting docker-registry.svc.toolforge.org proxy to use dns entry to floating ip instead === 2025-05-20 === * 19:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 19:40 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 19:39 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 19:39 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 17:18 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 17:18 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 17:17 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 17:16 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 17:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 16:11 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 16:11 wmbot~dcaro@acme: Updating container image docker-registry.svc.toolforge.org/toolforge-kyverno-kyverno:v1.13.6 * 16:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:48 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 15:48 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/busybox:1.35 * 15:47 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.30.2 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports:v1.13.6 * 15:46 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background:v1.13.6 * 15:45 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.13.6 * 15:44 wmbot~dcaro@acme: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.13.6 * 15:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 15:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 15:00 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 15:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:59 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 14:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 14:58 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=97) * 14:58 wmbot~dcaro@acme: Updating container image toolforge-kyverno-kyverno:v1.13.6 * 14:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 13:57 taavi: disable host-based authentication in sshd config, not used since grid shutdown * 13:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:07 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-prometheus-7 * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-prometheus-7 * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-prometheus-8.tools.eqiad1.wikimedia.cloud * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-19 === * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 17:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-9 * 16:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor ([[phab:T394520|T394520]]) * 16:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T394520|T394520]]) * 16:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-14 === * 17:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 * 07:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 === 2025-05-12 === * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 arturo: add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512 ([[phab:T393686|T393686]]) * 11:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 02:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 * 02:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 === 2025-05-10 === * 17:35 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo<nowiki>{</nowiki>,.socket<nowiki>}</nowiki> # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict ([[phab:T393732|T393732]]) * 17:34 lucaswerkmeister: root@tools-bastion-13:~# systemctl reset-failed sssd-<nowiki>{</nowiki>pam,sudo<nowiki>}</nowiki>.service && systemctl restart sssd-pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket # try to reset the rate limits this way ([[phab:T393732|T393732]]) * 16:22 lucaswerkmeister: systemctl restart sssd-<nowiki>{</nowiki>pam<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>,sudo<nowiki>}</nowiki>.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 14:10 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, [[phab:T393732|T393732]]? * 11:53 lucaswerkmeister: [[phab:T393732|T393732]] note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) * 11:52 lucaswerkmeister: root@tools-bastion-13:~# systemctl restart sssd-pam<nowiki>{</nowiki>,<nowiki>{</nowiki>,-priv<nowiki>}</nowiki>.socket<nowiki>}</nowiki> # all three failed with start-limit-hit / Start request repeated too quickly; [[phab:T393732|T393732]]? === 2025-05-09 === * 12:31 arturo: hard-reboot tools-bastion-13 (login.toolforge.org) because unresponsive (out of memory) -- previous reboot was for tools-bastion-12 (dev.t.o) by mistake * 12:29 arturo: hard-reboot tools-bastion-12 (login.toolforge.org) because unresponsive (out of memory) * 07:10 taavi: kill bunch of unwanted processes off of tools-bastion-13 [[phab:T393732|T393732]], please run your things as jobs === 2025-05-08 === * 17:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-admission * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:24 taavi: root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # was in failed state * 08:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-07 === * 18:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector-2 * 17:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector-2 * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:40 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:33 dcaro: released jobs-cli 16.1.12 * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 09:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 15:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 12:10 dcaro: rebooting tools-k8s-worker-nfs-69 due to some stuck processes * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 === 2025-05-04 === * 11:12 dcaro: deleting tools-services-05, has been off for a year (replaced with 06) === 2025-05-02 === * 18:37 taavi: add elasticsearch credential for tools.techcontribs [[phab:T393209|T393209]] * 13:55 taavi: reboot tools-static-15 === 2025-04-28 === * 13:07 dhinus: tools-db-4: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:06 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T392596|T392596]] * 13:05 dhinus: tools-db-5: systemctl stop mariadb && systemctl start mariadb [[phab:T318479|T318479]] === 2025-04-24 === * 23:09 bd808: `systemctl stop sssd; rm -rf /var/lib/sss/db/*; systemctl restart sssd` on tools-bastion-12 * 23:03 bd808: `sss_cache -E` on tools-bastion-12 after seeing "sudo: PAM account management error: Authentication service cannot retrieve authentication info" * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 18:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:51 taavi: add missing ICMPv6 security group rule to 'default' group * 08:02 taavi: add an AAAA record for toolserver.org [[phab:T392506|T392506]] === 2025-04-23 === * 19:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-3.tools.eqiad1.wikimedia.cloud * 15:10 arturo: give `tools-tofu` bot account member powers for https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:02 taavi: rebooting tools-mail-4 with stuck NFS handles === 2025-04-21 === * 09:52 taavi: update pywikibot-scripts-stable image to v10.0.0 [[phab:T385400|T385400]] === 2025-04-17 === * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-11 === * 21:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-10 === * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 === 2025-04-09 === * 21:35 bd808: Removed rook and sstefanova from https://gitlab.wikimedia.org/groups/toolforge-repos/ owners (both offboarded former WMCS staff) * 10:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 15:17 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 15:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 02:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-04-07 === * 19:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-109 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-79 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-78 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-77 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 13:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:58 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:56 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:54 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:52 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-111 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:49 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:48 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-110 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:47 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:43 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:37 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:30 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:15 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 12:07 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 11:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.28.14 to 1.29.15 ([[phab:T390214|T390214]]) * 08:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 05:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 05:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-06 === * 02:12 andrewbogott: truncating large logfiles on tools nfs === 2025-04-04 === * 10:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 08:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 07:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes === 2025-04-03 === * 22:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 22:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 22:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 22:22 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-33 * 22:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33 * 22:13 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 22:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 22:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-71 * 21:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-70, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-74 * 21:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 21:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 08:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 08:46 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 === 2025-04-02 === * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 20:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68, tools-k8s-worker-nfs-55 * 12:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 * 12:37 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 === 2025-04-01 === * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 13:59 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 13:56 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 13:55 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 13:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 === 2025-03-31 === * 12:48 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 12:42 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 12:03 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 * 11:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 * 09:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 08:59 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 === 2025-03-28 === * 16:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 16:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:58 taavi: reboot tools-static-15 due to stuck nginx worker processes * 10:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T389733|T389733]]) * 10:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T389733|T389733]]) * 09:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor ([[phab:T389733|T389733]]) * 09:30 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor ([[phab:T389733|T389733]]) === 2025-03-27 === * 17:34 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40, tools-k8s-worker-nfs-33 * 17:26 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:59 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:53 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all NFS workers * 15:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:02 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-111.tools.eqiad1.wikimedia.cloud to the cluster * 14:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 * 14:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17, tools-k8s-worker-nfs-21, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-72 === 2025-03-25 === * 15:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:18 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 * 13:58 andrewbogott: rebooting tools-k8s-worker-nfs-2 * 13:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 * 10:32 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 08:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 18:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 18:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:59 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 === 2025-03-22 === * 04:00 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 03:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 03:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 03:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-18 === * 15:23 arturo: hard-reboot tools-prometheus-6, not responding to ssh * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 10:30 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 10:03 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) * 09:57 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T383238|T383238]]) === 2025-03-17 === * 19:01 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 19:00 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 18:42 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:37 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 18:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 ([[phab:T383238|T383238]]) * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T388965|T388965]]) === 2025-03-16 === * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 === 2025-03-15 === * 15:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 ([[phab:T388965|T388965]]) * 12:55 dcaro: there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit [[phab:T388965|T388965]] === 2025-03-13 === * 22:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 18:00 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:50 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:40 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:27 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:17 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 16:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 16:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 16:25 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 16:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 10:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 === 2025-03-12 === * 17:56 dhinus: aptly repo remove bookworm-tools helmfile, removing custom version that is older than the one from apt.w.o * 03:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 17:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-03-07 === * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 13:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2025-03-06 === * 13:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 12:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-03-05 === * 19:16 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 (the two prom hosts are returning different values) * 17:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.2 ([[phab:T362868|T362868]]) * 17:44 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 16:06 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:13 dcaro: restarting ingress pods due to ingress timing out sometimes * 08:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 08:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-03-04 === * 20:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.12.0 ([[phab:T362868|T362868]]) * 14:01 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362868|T362868]]) * 13:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:40 dhinus: reboot tools-legacy-redirector-2 (http probes failing more than usual) * 12:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 09:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 09:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-03 === * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-03-01 === * 19:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 19:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 16:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 === 2025-02-27 === * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder === 2025-02-26 === * 14:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-25 === * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 === 2025-02-24 === * 21:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 21:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 21:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-21 === * 12:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 * 12:52 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 === 2025-02-20 === * 13:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-19 === * 20:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55 * 20:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 === 2025-02-18 === * 17:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 17:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-54 * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103, tools-k8s-worker-108, tools-k8s-control-7 ([[phab:T380679|T380679]]) * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 ([[phab:T380679|T380679]]) * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 ([[phab:T380679|T380679]]) === 2025-02-17 === * 17:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-10 === * 12:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-02-09 === * 16:38 andrewbogott: rebooting tools-db-4 just in case that helps with the recurring DB crashes === 2025-02-07 === * 20:51 arturo: resize tools-legacy-redirector to have 2 vCPU [[phab:T385908|T385908]] * 17:58 andrewbogott: "SET GLOBAL read_only=OFF; " on tools-db-4; both -5 and -4 were set to read only. No idea why or how... * 01:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 01:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-07 * 01:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-07 === 2025-02-06 === * 17:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:06 andrewbogott: cold-migrating tools-proxy-8 for [[phab:T385264|T385264]]; will cause a brief toolforge outage * 14:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-03 === * 14:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-haproxy-5, tools-k8s-haproxy-6 * 13:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9, tools-k8s-ingress-7, tools-k8s-ingress-8, tools-k8s-ingress-9 * 13:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 13:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 13:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 === 2025-02-01 === * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-108 * 15:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-107 * 15:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-106 * 15:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-105 * 15:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-103 * 15:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-102 * 15:01 andrewbogott: rebooting all k8s (non-nfs) worker nodes for [[phab:T385264|T385264]] * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-102 * 14:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 * 14:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-71 * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-66 * 14:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 * 14:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-46 * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-46 * 14:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 * 14:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-40 * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-39 * 14:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 14:37 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 * 14:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 14:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 * 14:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 * 14:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 * 14:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-11 * 14:29 andrewbogott: rebooting all k8s-nfs worker nodes for [[phab:T385264|T385264]] * 14:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-11 * 14:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 * 14:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-10 * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-10 === 2025-01-31 === * 11:04 dhinus: systemctl restart prometheus@tools on tools-prometheus-7 [[phab:T385262|T385262]] === 2025-01-29 === * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-27 === * 16:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 15:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-26 === * 22:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 22:04 andrewbogott: restarting Node tools-k8s-worker-nfs-44 , too many D processes * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-m8s-worker-nfs-44 * 22:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-m8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-79.tools.eqiad1.wikimedia.cloud to the cluster * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-78.tools.eqiad1.wikimedia.cloud to the cluster * 08:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-77.tools.eqiad1.wikimedia.cloud to the cluster * 08:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T384790|T384790]]) * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 08:06 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-110.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 07:56 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-109.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster ([[phab:T384790|T384790]]) * 07:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 * 07:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-55 === 2025-01-24 === * 10:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 * 10:34 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 === 2025-01-23 === * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:10 dcaro: reboot tools-static-15 due to nginx stuck on nfs === 2025-01-22 === * 17:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2025-01-18 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 15:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2025-01-17 === * 15:52 dhinus: reboot tools-legacy-redirector-2 (http probes were failing) === 2025-01-15 === * 04:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-13 === * 21:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:31 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383625|T383625]]) * 21:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 ([[phab:T383238|T383238]]) * 21:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 ([[phab:T383625|T383625]]) * 21:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 21:18 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383625|T383625]]) * 21:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:14 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T383238|T383238]]) * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 ([[phab:T383238|T383238]]) * 21:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 ([[phab:T383625|T383625]]) * 21:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 21:03 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 ([[phab:T383238|T383238]]) * 20:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 ([[phab:T383238|T383238]]) * 20:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 ([[phab:T383625|T383625]]) * 20:49 dcaro: restart prometheus to pick up the new ips for vms and such * 20:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 20:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 * 20:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 ([[phab:T383625|T383625]]) * 20:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383625|T383625]]) * 20:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 ([[phab:T383238|T383238]]) * 20:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:41 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 * 20:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 20:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 20:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 20:36 lucaswerkmeister: restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots === 2025-01-12 === * 09:53 taavi: hard reboot tools-k8s-worker-nfs-55 === 2025-01-08 === * 18:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-43 ([[phab:T383238|T383238]]) * 18:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-32 ([[phab:T383238|T383238]]) * 18:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 ([[phab:T383238|T383238]]) * 18:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 ([[phab:T383238|T383238]]) * 18:12 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 ([[phab:T383238|T383238]]) * 18:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-41 ([[phab:T383238|T383238]]) * 18:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 ([[phab:T383238|T383238]]) * 17:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-27 ([[phab:T383238|T383238]]) * 17:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 ([[phab:T383238|T383238]]) * 17:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 ([[phab:T383238|T383238]]) * 17:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-26 ([[phab:T383238|T383238]]) * 17:34 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-76 ([[phab:T383238|T383238]]) * 17:27 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 ([[phab:T383238|T383238]]) * 17:14 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-12 ([[phab:T383238|T383238]]) * 17:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-48 ([[phab:T383238|T383238]]) * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-57 ([[phab:T383238|T383238]]) * 16:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-65 ([[phab:T383238|T383238]]) * 16:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 ([[phab:T383238|T383238]]) * 16:25 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 ([[phab:T383238|T383238]]) * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 ([[phab:T383238|T383238]]) * 15:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 * 15:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-42 * 15:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22 * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22 * 15:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 14:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-70 * 14:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-70 * 14:16 dcaro: reboot tools-static-15 nfs is stuck === 2025-01-07 === * 00:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor === 2025-01-06 === * 23:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 23:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-01-05 === * 18:58 lucaswerkmeister: remove /tmp/framer.txt on tools-bastion-13 (I notified the owner privately), and replace it with a root-owned file to prevent iTerm from leaking logs into it (https://iterm2.com/downloads/stable/iTerm2-3_5_11.changelog) on tools-sgebastion-10, tools-bastion-12 and tools-bastion-13 === 2025-01-03 === * 21:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-69 * 21:41 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-69 * 21:40 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-69 * 21:35 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-69 === 2025-01-02 === * 02:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 02:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 === 2025-01-01 === * 21:10 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 21:05 andrewbogott: truncating *.err and *.out files to clear out NFS space * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 * 21:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-34 * 20:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-34 === 2024-12-13 === * 14:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 14:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-68 * 09:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-68 * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 * 09:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-44 * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-73 * 08:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-73 === 2024-12-12 === * 10:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-5 * 10:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-5 === 2024-12-06 === * 17:26 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:25 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-1 ([[phab:T352206|T352206]]) * 17:24 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-3 ([[phab:T352206|T352206]]) * 17:23 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-3 ([[phab:T352206|T352206]]) * 07:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 19:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 19:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:46 andrewbogott: rebooting tools-legacy-redirector-2, many probes failing * 17:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:45 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:11 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:46 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 22:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component main * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component main === 2024-11-29 === * 03:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-27 === * 18:26 taavi: kubectl sudo rollout restart -n kube-system deployment coredns # update resolv.conf in coredns containers === 2024-11-26 === * 10:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-7 * 10:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-control-7 * 10:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-7 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-9 * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-9 * 10:22 dcaro: rebooting k8s-control-9 * 10:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-control-8 * 10:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-control-8 * 10:17 dcaro: rebooting k8s-control-8 * 09:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 * 09:14 dcaro: restarting tools-k8s-worker-nfs-72 * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 * 09:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 09:12 dcaro: restarting tools-k8s-worker-nfs-70 * 09:11 dcaro: restarting tools-k8s-worker-nfs-50 * 09:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 09:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 08:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-61 * 08:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-61 * 07:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T380827|T380827]]) * 06:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T380827|T380827]]) === 2024-11-25 === * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2024-11-23 === * 07:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 21:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 21:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 20:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 20:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 20:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component envvars-api ([[phab:T362867|T362867]]) * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api ([[phab:T362867|T362867]]) * 20:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 20:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-18 === * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-15 === * 14:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:04 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-5.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T352206|T352206]]) * 13:57 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T352206|T352206]]) * 13:50 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 13:49 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) === 2024-11-14 === * 13:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 13:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 15:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 16:02 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 15:58 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:44 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:42 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-db-4.tools.eqiad1.wikimedia.cloud ([[phab:T352206|T352206]]) * 14:41 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:37 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-db' ([[phab:T352206|T352206]]) * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-10 === * 02:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.11.0 ([[phab:T362867|T362867]]) * 02:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T362867|T362867]]) === 2024-11-06 === * 16:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 ([[phab:T379139|T379139]]) * 07:57 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:30 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:22 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 16:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-76 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-75 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-74 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-73 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-72 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-71 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-70 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-69 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-68 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-67 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-66 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-65 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 12:11 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 12:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 11:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 dcaro: added api.svc.toolforge.org dns record entry * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:56 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:51 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 09:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-22 === * 13:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 * 12:58 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-33, tools-k8s-woker-nfs-23 * 09:05 arturo: restart puppetserver service for [[phab:T377803|T377803]] === 2024-10-16 === * 09:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-14 === * 09:14 dcaro: migrating pipelineruns stored versions to v1 ([[phab:T376710|T376710]]) * 07:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 07:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 * 07:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-10-09 === * 09:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:38 dcaro: tests are passing correctly, upgrade finished, will investigate the increased slowness as a followup * 12:27 dcaro: upgrade finished, build actions have become slower than usual ([[phab:T376710|T376710]]), running tests and investigating * 12:02 dcaro: starting toolforge builds-builder upgrade, no downtime expected though some builds might fail to start/list/log/show while the upgrade is in progress [[phab:T374908|T374908]] * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-04 === * 11:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-02 === * 09:11 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-01 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:28 dcaro: updated ci image with latest precommit versions * 10:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-30 === * 18:25 taavi: run striker migrations [[phab:T359428|T359428]] === 2024-09-28 === * 00:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:58 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 16:08 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:05 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 15:58 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:20 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 10:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:53 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 08:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 * 07:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 === 2024-09-24 === * 22:11 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 22:03 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:48 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-20 === * 20:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 20:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 20:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 17:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:06 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/pod2daemon-flexvol:v3.28.2 ([[phab:T359641|T359641]]) * 17:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/typha:v3.28.2 ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/node:v3.28.2 ([[phab:T359641|T359641]]) * 17:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/kube-controllers:v3.28.2 ([[phab:T359641|T359641]]) * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/ctl:v3.28.2 ([[phab:T359641|T359641]]) * 16:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/calico/cni:v3.28.2 ([[phab:T359641|T359641]]) * 16:54 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 06:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 00:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:32 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 23:17 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.10 ([[phab:T359641|T359641]]) * 23:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 23:12 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.10.1 ([[phab:T359641|T359641]]) * 23:11 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:38 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=97) ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/docker-registry.tools.wmflabs.org/metrics-server:v0.7.1 ([[phab:T359641|T359641]]) * 22:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 17:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:13 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 17:06 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:48 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 16:46 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:45 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 16:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:38 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 16:10 dcaro: rebooting tools-k8s-worker-nfs-24 it's stuck without network * 16:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:08 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:07 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:28 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:19 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:18 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:08 wmbot~raymondndibe@wmf3402: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api ([[phab:T341066|T341066]]) * 15:07 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 15:01 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:57 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:56 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-17 === * 08:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-70 ([[phab:T359641|T359641]]) * 08:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 ([[phab:T359641|T359641]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 08:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud ([[phab:T359641|T359641]]) * 03:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 03:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-64 * 03:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-63 * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 03:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:07 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-76.tools.eqiad1.wikimedia.cloud to the cluster * 03:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 03:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 03:00 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-75.tools.eqiad1.wikimedia.cloud to the cluster * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:46 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-74.tools.eqiad1.wikimedia.cloud to the cluster * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-62 * 02:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-60 * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 02:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:38 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-73.tools.eqiad1.wikimedia.cloud to the cluster * 02:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-72.tools.eqiad1.wikimedia.cloud to the cluster * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:24 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-71.tools.eqiad1.wikimedia.cloud to the cluster * 02:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:12 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-6 * 02:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-56 * 02:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 02:08 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-70.tools.eqiad1.wikimedia.cloud to the cluster * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 02:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-49 * 02:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-31 * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:57 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-69.tools.eqiad1.wikimedia.cloud to the cluster * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-30 * 01:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-29 * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-64 ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-28 * 01:42 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:42 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-68.tools.eqiad1.wikimedia.cloud to the cluster * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-62 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:32 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-67.tools.eqiad1.wikimedia.cloud to the cluster * 01:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 01:23 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-66.tools.eqiad1.wikimedia.cloud to the cluster * 01:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-49 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 01:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-31 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-30 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50, tools-k8s-worker-nfs-56, tools-k8s-worker-nfs-57, tools-k8s-worker-nfs-6 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38, tools-k8s-worker-nfs-46, tools-k8s-worker-nfs-49, tools-k8s-worker-nfs-50 ([[phab:T359641|T359641]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-60, tools-k8s-worker-nfs-61, tools-k8s-worker-nfs-62, tools-k8s-worker-nfs-63 ([[phab:T359641|T359641]]) * 00:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-31, tools-k8s-worker-nfs-32, tools-k8s-worker-nfs-33, tools-k8s-worker-nfs-36 ([[phab:T359641|T359641]]) === 2024-09-16 === * 17:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 * 17:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 * 17:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 17:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-09-13 === * 11:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-54 ([[phab:T374692|T374692]]) * 09:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-55, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-14 ([[phab:T374692|T374692]]) === 2024-09-12 === * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:54 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23, tools-k8s-worker-16, tools-k8s-worker-nfs-33 ([[phab:T374612|T374612]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) * 11:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-28 ([[phab:T374612|T374612]]) === 2024-09-11 === * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-09 === * 16:23 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager === 2024-09-06 === * 08:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:38 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 07:14 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/pause:3.6 * 07:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-09-05 === * 13:50 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/stakater-reloader:v1.1.0 ([[phab:T359641|T359641]]) * 13:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:45 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:41 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=99) ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/startupapicheck:v1.15.3 ([[phab:T359641|T359641]]) * 13:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:28 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/cainjector:v1.15.3 ([[phab:T359641|T359641]]) * 13:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/webhook:v1.15.3 ([[phab:T359641|T359641]]) * 13:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) * 13:24 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: Updating container image docker-registry.tools.wmflabs.org/cert-manager/controller:v1.15.3 ([[phab:T359641|T359641]]) * 13:23 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 13:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 13:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-03 === * 20:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:53 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:36 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 15:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:05 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno-cli:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 14:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) ([[phab:T359641|T359641]]) * 13:55 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.28.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:54 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.12.5 ([[phab:T359641|T359641]]) * 13:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry ([[phab:T359641|T359641]]) * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.25.16 to 1.26.15 * 05:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.25.16 to 1.26.15 * 05:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 * 05:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.25.16 to 1.26.15 === 2024-09-02 === * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-64 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 14:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.25.16 to 1.26.15 * 13:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-63 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.25.16 to 1.26.15 * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-62 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.25.16 to 1.26.15 * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-61 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-60 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-58 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.25.16 to 1.26.15 * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-57 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.25.16 to 1.26.15 * 13:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.25.16 to 1.26.15 * 13:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.25.16 to 1.26.15 * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.25.16 to 1.26.15 * 13:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.25.16 to 1.26.15 * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.25.16 to 1.26.15 * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.25.16 to 1.26.15 * 13:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.25.16 to 1.26.15 * 13:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.25.16 to 1.26.15 * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.25.16 to 1.26.15 * 13:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 13:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.25.16 to 1.26.15 * 12:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.25.16 to 1.26.15 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.25.16 to 1.26.15 * 12:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.25.16 to 1.26.15 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.25.16 to 1.26.15 ([[phab:T370249|T370249]]) * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.25.16 to 1.26.15 * 12:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.25.16 to 1.26.15 * 12:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.25.16 to 1.26.15 * 12:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.25.16 to 1.26.15 * 12:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 12:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.25.16 to 1.26.15 * 12:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.25.16 to 1.26.15 * 11:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.25.16 to 1.26.15 * 11:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.25.16 to 1.26.15 * 11:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.25.16 to 1.26.15 * 10:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:43 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:48 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 07:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 12:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:06 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.11.2 * 12:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 09:46 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 wmbot~dcaro@urcuchillay: Added a new k8s worker tools-k8s-worker-108.tools.eqiad1.wikimedia.cloud to the cluster * 09:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-52 ([[phab:T373243|T373243]]) * 08:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-51 ([[phab:T373243|T373243]]) * 08:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-25 ([[phab:T373243|T373243]]) * 08:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-18 ([[phab:T373243|T373243]]) * 08:29 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-15 ([[phab:T373243|T373243]]) * 08:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 08:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-08-26 === * 21:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 21:13 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-64.tools.eqiad1.wikimedia.cloud to the cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 21:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 20:23 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-63.tools.eqiad1.wikimedia.cloud to the cluster * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 20:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 20:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 18:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:49 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-62.tools.eqiad1.wikimedia.cloud to the cluster * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 17:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.quota_increase * 17:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 17:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 17:04 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-61.tools.eqiad1.wikimedia.cloud to the cluster * 16:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:54 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-60.tools.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 16:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:14 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-58.tools.eqiad1.wikimedia.cloud to the cluster * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 16:02 wmbot~dcaro@urcuchillay: Added a new k8s worker-nfs tools-k8s-worker-nfs-57.tools.eqiad1.wikimedia.cloud to the cluster * 15:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:44 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:38 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the tools cluster * 15:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 15:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 ([[phab:T373243|T373243]]) * 13:12 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 13:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-4, tools-k8s-worker-nfs-15, tools-k8s-worker-nfs-18, tools-k8s-worker-nfs-25, tools-k8s-worker-nfs-51, tools-k8s-worker-nfs-52, tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-104 ([[phab:T373243|T373243]]) * 11:06 dcaro: manually deleted the coredns pods that had been around for 4d * 09:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 08:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:18 dcaro: scale up cordens deployment to 4 replicas === 2024-08-21 === * 05:44 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:27 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:43 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:28 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:25 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:22 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:20 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 04:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 04:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:49 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:28 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:19 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 03:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 03:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-19 === * 22:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-24 * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-24 * 21:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17 * 21:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 * 21:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-17,tools-k8s-worker-nfs-24 === 2024-08-15 === * 06:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-20 * 06:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 === 2024-08-13 === * 09:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-6 * 07:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-6 === 2024-08-12 === * 15:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-06 === * 09:50 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=1) * 09:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:20 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:19 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2024-08-05 === * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 20:42 bd808: Uncordoned tools-k8s-worker-nfs-55 following reboot * 20:40 bd808: Hard reboot of tools-k8s-worker-nfs-55 following drain cookbook run. Stuck pod remained stuck as expected. * 20:37 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-55 * 20:32 bd808: Draining and rebooting tools-k8s-worker-nfs-55 after reports of stuck pods via irc * 20:32 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 15:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 20:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 16:17 andrewbogott: changing login.tools.wmlabs.org to point to a newer bastion, tools-bastion-12, in response to [[phab:T371505|T371505]] * 11:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 * 09:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-26, tools-k8s-worker-nfs-43 === 2024-07-30 === * 18:08 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 18:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 18:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 18:02 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-cli * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:40 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 17:39 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 17:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-23 * 16:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-23 === 2024-07-29 === * 18:24 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:06 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:05 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.rebuild_dbinstance (exit_code=0) * 14:03 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.rebuild_dbinstance * 13:19 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 13:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 12:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 12:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 12:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 09:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 08:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 07:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component ingress-admission * 06:57 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 15:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 15:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 13:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 17:42 dcaro: moved the apt repo to service endpoint deb.svc.toolforge.org * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-3 * 17:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-3 * 17:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 17:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 17:00 dcaro: moving the toolforge apt repo to tools-services-06 * 16:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-services-06.tools.eqiad1.wikimedia.cloud * 16:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-services-06.tools.eqiad1.wikimedia.cloud * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-19 === * 12:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:46 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.9.2 * 12:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 10:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 10:02 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/nginx-ingress-controller:v1.9.6 * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry === 2024-07-18 === * 14:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:12 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.24.17 to 1.25.16 * 14:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.24.17 to 1.25.16 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.24.17 to 1.25.16 * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.24.17 to 1.25.16 * 11:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.24.17 to 1.25.16 * 11:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.24.17 to 1.25.16 * 11:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.24.17 to 1.25.16 * 11:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 11:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.24.17 to 1.25.16 * 11:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.24.17 to 1.25.16 * 11:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.24.17 to 1.25.16 * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.24.17 to 1.25.16 * 11:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.24.17 to 1.25.16 * 11:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.24.17 to 1.25.16 * 11:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-nfs-worker-21 from 1.24.17 to 1.25.16 * 11:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 10:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.24.17 to 1.25.16 * 10:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:57 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.24.17 to 1.25.16 * 10:55 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.24.17 to 1.25.16 * 10:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.24.17 to 1.25.16 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.24.17 to 1.25.16 * 10:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.24.17 to 1.25.16 * 10:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.24.17 to 1.25.16 * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.24.17 to 1.25.16 * 10:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.24.17 to 1.25.16 * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.24.17 to 1.25.16 * 10:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.24.17 to 1.25.16 * 10:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.24.17 to 1.25.16 * 10:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.24.17 to 1.25.16 * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.24.17 to 1.25.16 * 10:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.24.17 to 1.25.16 * 10:41 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.24.17 to 1.25.16 * 10:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.24.17 to 1.25.16 * 10:40 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.24.17 to 1.25.16 * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.24.17 to 1.25.16 * 10:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:39 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.24.17 to 1.25.16 * 10:38 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.24.17 to 1.25.16 * 10:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.24.17 to 1.25.16 * 10:37 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.24.17 to 1.25.16 * 10:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.24.17 to 1.25.16 * 10:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.24.17 to 1.25.16 * 10:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.24.17 to 1.25.16 * 10:31 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.24.17 to 1.25.16 * 10:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.24.17 to 1.25.16 * 10:27 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.24.17 to 1.25.16 * 10:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.24.17 to 1.25.16 * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.24.17 to 1.25.16 * 10:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.24.17 to 1.25.16 * 10:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:20 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.24.17 to 1.25.16 * 10:19 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.24.17 to 1.25.16 * 10:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.24.17 to 1.25.16 * 10:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 10:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 10:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 10:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.24.17 to 1.25.16 * 10:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.24.17 to 1.25.16 * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.24.17 to 1.25.16 * 09:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-1 from 1.24.17 to 1.25.16 * 09:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.24.17 to 1.25.16 * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.24.17 to 1.25.16 * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.24.17 to 1.25.16 * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.24.17 to 1.25.16 * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.24.17 to 1.25.16 * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:42 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-11 === * 17:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:49 dcaro: deploy toolforge-jobs-framework 16.0.13 ([[phab:T369573|T369573]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 20:22 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-37 * 20:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-37 * 14:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-3 * 13:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-2 * 13:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-elastic-1 * 13:56 andrew@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-elastic-1 * 13:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 12:29 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:27 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:27 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 12:26 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 12:26 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 12:23 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) * 12:23 sstefanova@cloudcumin1001: Updating container image docker-registry.tools.wmflabs.org/kube-state-metrics:v2.7.0 * 12:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry * 11:29 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.image.copy_to_registry (exit_code=0) copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 11:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.image.copy_to_registry copy image from bitnami/kubectl:1.26.4 to docker-registry.tools.wmflabs.org/bitnami-kubectl:1.26.4 * 01:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:46 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 12:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:16 andrewbogott: draining (I hope) tools-elastic-3 and tools-elastic-1 for [[phab:T311905|T311905]] * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:53 arturo: cleanup kubeadm configmap from TTLAfterFinished settings ([[phab:T349197|T349197]]) * 11:51 arturo: remove --feature-gates=TTLAfterFinished=true from kube-controller-manager static pod definition ([[phab:T349197|T349197]]) * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:38 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-23 * 16:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-23 * 16:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-1 * 16:21 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-db-1 * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-1 * 15:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-db-3 * 15:46 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-db-3 * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-24 * 15:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-24 * 15:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-etcd-22 * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-etcd-22 * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 11:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 arturo: drop all PSP definitions for all accounts ([[phab:T368142|T368142]]) * 10:02 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 11:40 taavi: update pywikibot image to 9.2 [[phab:T363631|T363631]] * 10:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:18 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:18 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-29 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-29 === 2024-06-25 === * 21:50 bd808: Live hacked /usr/lib/python3/dist-packages/toolsws/backends/kubernetes.py on login-buster.toolforge.org to remove the `-> dict[str, Any]` type annotations causing [[phab:T368463|T368463]] * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-104 * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-104 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-103 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-104 * 12:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-103 * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-102 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-103 * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-56 * 12:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-56 * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-55 * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-55 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-54 * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-56 * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-54 * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-53 * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-55 * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-53 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-52 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-54 * 12:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-52 * 12:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-51 * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-53 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-51 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-53 * 11:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-52 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-50 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-52 * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-50 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 11:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-50 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-50 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-50 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-7 * 11:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-7 * 11:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.11 to server 'tools-proxy-8' * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.11 to server 'tools-proxy-8' * 09:44 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-6 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-9 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-9 * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-9 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-9 * 08:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-49 * 08:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-49 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-48 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-49 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-47 * 08:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-48 * 08:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-47 * 08:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-46 * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-45 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-47 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-47 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-45 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-44 * 08:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-46 * 08:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-46 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-44 * 08:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-45 * 08:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-k8s-worker-nfs-43 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-42 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-44 * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-44 * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-42 * 08:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:07 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-42 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-41 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-42 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-41 * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-40 * 07:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-39 * 07:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-41 * 07:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-39 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-38 * 07:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-40 * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-38 * 07:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-37 * 07:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-39 * 07:55 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-37 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-36 * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 07:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-36 * 07:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-35 * 07:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-37 * 07:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-35 * 07:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-34 * 07:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-36 * 07:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-34 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-35 * 07:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-33 * 07:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-35 * 07:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-34 * 07:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-33 * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-33 * 07:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-33 === 2024-06-24 === * 20:56 andrewbogott: rebooting tools-k8s-worker-nfs-36; it has lots of stuck processes which somehow didn't get unstuck when we did the post-nfs-migration reboots. * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-32 * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-32 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-31 * 15:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-32 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-31 * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-32 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-30 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-31 * 15:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-30 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-29 * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-30 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-29 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-28 * 15:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-29 * 15:45 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-29 * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-28 * 15:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-28 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-27 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-28 * 15:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-27 * 15:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-27 * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-sgebastion-10 * 14:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-sgebastion-10 * 14:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-13 * 14:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-13 * 14:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-bastion-12 * 14:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 14:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-nfs-2 * 14:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server tools-nfs-2 * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-nfs-2 * 13:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd * 13:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-26 * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-25 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-26 * 13:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-24 * 13:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-26 * 13:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-24 * 13:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-24 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-23 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-24 * 13:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-22 * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-22 * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-21 * 13:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-23 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-21 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-20 * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-22 * 13:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-20 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-21 * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-19 * 13:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-21 * 13:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-19 * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-18 * 13:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-20 * 13:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-17 * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-20 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-19 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-18 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-17 * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-17 * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-16 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-16 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-15 * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-16 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-15 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-14 * 12:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-15 * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-14 * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-13 * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-14 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-13 * 12:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-12 * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-13 * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-12 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-12 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-11 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-12 * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-7 * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-11 * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-7 * 12:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-8 * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-8 * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-8 * 12:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-8 * 12:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-static-15 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-static-15 * 12:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-acme-chief-4 * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-acme-chief-4 * 12:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=97) for node tools-k8s-worker-nfs-10 * 11:58 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-10 * 11:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:56 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-10 * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-10 * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-9 * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-9 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-8 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-8 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-8 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-7 * 11:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-8 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-7 * 11:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-7 * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-6 * 11:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-5 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-4 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-6 * 11:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-4 * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-5 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-4 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-4 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-3 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-3 * 11:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-1 * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-3 * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-2 * 11:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-2 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-1 * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-5 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-5 * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-7 * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-7 * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-ingress-7 * 10:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-worker-nfs-43 * 10:11 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-ingress-7 * 10:09 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-worker-nfs-43 * 10:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-control-7 * 10:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-control-7 * 10:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-7 * 10:03 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-43 * 10:02 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-7 * 10:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-redis-6 * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-redis-6 * 09:58 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-43 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-cumin-1 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-cumin-1 * 09:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-k8s-haproxy-5 * 09:50 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-k8s-haproxy-5 * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-harbor-1 * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-harbor-1 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-107.tools.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-prometheus-6 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-prometheus-6 * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetserver-01 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetserver-01 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-puppetdb-2 * 09:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-puppetdb-2 * 09:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:30 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-106.tools.eqiad1.wikimedia.cloud to the cluster * 09:30 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-mail-4 * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-legacy-redirector-2 * 09:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-legacy-redirector-2 * 09:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-imagebuilder-2 * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-imagebuilder-2 * 09:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-proxy-8 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-services-05 * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-services-05 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-package-builder-04 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-docker-registry-8 * 09:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:19 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-docker-registry-8 * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server tools-checker-5 * 09:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 09:18 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-105.tools.eqiad1.wikimedia.cloud to the cluster * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server tools-checker-5 * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 09:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster === 2024-06-20 === * 13:09 arturo: re-deploy kyverno [[phab:T368044|T368044]] * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:11 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-14 === * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:15 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 19:41 bd808: Rebuilding all shared Docker containers. This will among other things apply the fix for [[phab:T367345|T367345]]. * 17:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 13:45 taavi: hard reboot tools-k8s-control-7 * 12:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 17:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers * 16:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 15:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all NFS workers * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:57 dcaro: cleaning old maintain-kubeusers configmaps * 10:45 dcaro: cleaning up old resourcequotas === 2024-06-10 === * 09:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno === 2024-06-07 === * 10:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 dcaro: deploying toolforge-webservice 0.103.6 * 12:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 08:44 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-13 * 08:41 dcaro: deploying toolforge-jobs-framework-cli 16.0.10 on tools-bastion-12 === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:16 wmbot~arturo@nostromo: END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 * 10:15 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 * 10:14 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:14 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 10:13 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 10:13 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 10:13 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:37 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:37 wmbot~arturo@nostromo: Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * 09:37 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:29 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:29 wmbot~arturo@nostromo: END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) * 09:28 wmbot~arturo@nostromo: START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry * 09:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-05-29 === * 16:14 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:13 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 02:59 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component envvars-api * 02:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-28 === * 10:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 15:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 09:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-25 === * 21:33 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:32 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:38 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-22 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 === 2024-05-15 === * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 14:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 ([[phab:T364822|T364822]]) * 10:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 07:48 dcaro: draining tools-k8s-worker-nfs-9 as it's stuck on IO * 07:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=99) for node tools-k8s-worker-nfs-9 * 07:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-9 === 2024-05-07 === * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 12:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-05 === * 07:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:48 taavi: update pywikibot script image to v9.1.0 [[phab:T363132|T363132]] === 2024-04-24 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-17 === * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-50 * 20:48 andrewbogott: In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-50 * 15:21 dcaro: swapped login.toolforge.org to point to tools-bastion-13 * 10:48 dcaro: rebooting tools-k8s-worker-nfs-1 === 2024-04-16 === * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-1 * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-1 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'python3-toolforge-weld' version '1.5.0' * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'python3-toolforge-weld' version '1.5.0' === 2024-04-15 === * 20:34 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:33 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:27 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 10:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:13 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:11 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-11 === * 08:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-09 === * 17:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:23 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 14:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:43 dcaro: deployed builds-builder 0.0.94 and removed builds-admission * 13:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:19 dcaro: deploying toolforge-jobs-cli 16.0.6 === 2024-04-08 === * 16:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:09 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:07 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:32 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 * 14:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 * 13:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-56 * 13:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-56 * 13:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:45 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:31 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:29 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 13:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 13:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:55 dcaro_: deploy toolforge-jobs-framework-cli 16.0.5 === 2024-04-05 === * 12:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 15:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 15:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:59 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:58 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:58 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:57 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:57 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:49 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:37 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:37 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:24 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:24 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:23 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-06 * 11:21 wmbot~taavi@runko: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-06 * 09:45 taavi: rebuilding prebuild images for [[phab:T361457|T361457]] === 2024-04-02 === * 12:39 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-db-2 ([[phab:T344717|T344717]]) * 12:38 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-db-2 ([[phab:T344717|T344717]]) * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-05 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-05 === 2024-03-28 === * 14:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-proxy-05 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-proxy-05 * 13:45 taavi: migrating toolforge.org floating IP from tools-proxy-06 to tools-proxy-7 [[phab:T361223|T361223]] * 13:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 13:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-proxy' * 13:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-proxy' * 12:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-registry-06 * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-registry-06 * 11:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' === 2024-03-27 === * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolserver-proxy-01 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolserver-proxy-01 === 2024-03-26 === * 16:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:41 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud * 16:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' * 16:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud * 12:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion' * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion' * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11 * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11 * 10:24 taavi: point toolserver.org DNS to tools-legacy-redirector-2 [[phab:T311909|T311909]] === 2024-03-25 === * 18:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-legacy-redirector * 18:23 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-legacy-redirector * 14:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud * 14:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-legacy-redirector-2.tools.eqiad1.wikimedia.cloud === 2024-03-22 === * 11:43 dcaro: restarted sssd on tools-prometheus-6 as it was stopped (error) === 2024-03-21 === * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-4 * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-4 * 15:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node tools-k8s-haproxy-3 * 15:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node tools-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=99) for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-k8s-haproxy-3 * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 12:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-checker-04 * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-checker-04 * 12:30 taavi: move checker service address to tools-checker-5 * 11:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:40 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-checker-5.tools.eqiad1.wikimedia.cloud * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-checker' * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 10:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'tools-checker' * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-checker' === 2024-03-19 === * 21:28 taavi: kick off full container image rebuild for https://gerrit.wikimedia.org/r/1012753 (python3 backwards compat in lighttpd images) and https://gerrit.wikimedia.org/r/1010690 (add procps to base images) * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-static-14 * 11:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-static-14 * 11:19 taavi: point dev.toolforge.org to tools-bastion-12 [[phab:T314665|T314665]] * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:38 dcaro: pushed docker-registry.tools.wmflabs.org/cloud-cicd-py311bookworm-tox:latest and docker-registry.tools.wmflabs.org/cloud-cicd-debian-builder-bookworm:2024-03-24.1 ([[phab:T360405|T360405]]) === 2024-03-18 === * 13:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 taavi: restart harbor services after docker service restart * 13:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-104 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:03 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-56 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-55 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-54 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-53 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-52 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-51 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-50 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-49 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-48 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-47 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-46 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-45 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-44 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-43 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-42 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-41 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-40 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-39 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:34 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-filesystemtest-1 * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-38 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-filesystemtest-1 * 12:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-37 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-36 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-35 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:29 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-34 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-33 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-32 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:26 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-31 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-30 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-29 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-28 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-27 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-26 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-25 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:19 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-24 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-23 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-4.tools.eqiad1.wikimedia.cloud * 12:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-22 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:11 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-21 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 12:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-acme-chief-3.tools.eqiad1.wikimedia.cloud * 11:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-20 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-19 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-18 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:52 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-17 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-16 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-15 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-14 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-13 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-9 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-8 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-7 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-6 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:39 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-5 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:23 taavi: point tools-static proxy to tools-static-15 (bookworm) [[phab:T311913|T311913]] * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-static-15.tools.eqiad1.wikimedia.cloud * 11:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 11:00 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 11:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster tools upgrade from 1.23.17 to 1.24.17 ([[phab:T307651|T307651]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-12.tools.eqiad1.wikimedia.cloud * 09:27 taavi: deleted shutdown grid engine VMs [[phab:T314664|T314664]] === 2024-03-15 === * 10:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'misctools' version '1.48' * 17:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'misctools' version '1.48' * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-docker-imagebuilder-01 * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 15:10 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.remove_instance (exit_code=99) for instance tools-docker-imagebuilder-01 * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-docker-imagebuilder-01 * 11:02 taavi: stop grid related VMs [[phab:T314664|T314664]] * 11:01 taavi: disable grid access for remaining tools still running on the grid [[phab:T314664|T314664]] === 2024-03-13 === * 19:21 andrewbogott: shutting down old puppet infra: tools-puppetmaster-02 and tools-puppetdb-1. These can be deleted in a week or two presuming everything remains stable. === 2024-03-12 === * 12:38 taavi: hard reboot tools-prometheus-6 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-11 === * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:20 arturo: cached registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.6.0 as docker-registry.tools.wmflabs.org/kube-state-metrics:v2.6.0 in the docker registry for [[phab:T359798|T359798]] === 2024-03-09 === * 12:48 taavi: hard reboot tools-sgebastion-10 due to stuck NFS procs === 2024-03-08 === * 12:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-07 === * 14:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:42 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-06 === * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_grid_node (exit_code=1) for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-32 * 10:34 taavi: rebuilding all docker images for https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/1005952 ([[phab:T293552|T293552]]) + normal package updates * 09:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 09:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:42 taavi: reboot tools-sgeexec-10-20, -21, -23, sgeweblight-10-32 due to stuck nfs procs === 2024-03-05 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 16:07 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 16:06 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) ([[phab:T357901|T357901]]) * 16:06 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 16:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud === 2024-03-04 === * 17:56 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:56 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:57 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:43 taavi: reboot tools-sgegrid-shadow due to high number of procs in D state === 2024-03-03 === * 10:38 dcaro: reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state) === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-29 === * 14:36 dcaro: deploy webservice 0.103.3 === 2024-02-28 === * 11:57 dcaro: deploy tools-webservice 0.103.2 with probes ([[phab:T341919|T341919]]) * 00:46 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:46 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 09:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 09:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 09:35 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:35 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) === 2024-02-23 === * 14:19 taavi: remove isc-dhcp-server (server, not client) from tools-db-2 * 13:32 taavi: remove toolschecker alerts for grid engine jobs [[phab:T358333|T358333]] === 2024-02-22 === * 14:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:17 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 14:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) ([[phab:T284656|T284656]]) * 11:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node ([[phab:T284656|T284656]]) * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 11:15 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-104.tools.eqiad1.wikimedia.cloud to the cluster * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:39 aborrero@cloudcumin1001: Added a new k8s control tools-k8s-control-8.tools.eqiad1.wikimedia.cloud to the cluster * 09:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster ([[phab:T284656|T284656]]) * 08:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-51 * 08:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-38 * 08:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-38 * 08:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-nfs-25 * 08:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-nfs-25 === 2024-02-21 === * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-control-4 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-control-4 * 09:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the tools cluster * 09:20 taavi@cloudcumin1001: Added a new k8s control tools-k8s-control-7.tools.eqiad1.wikimedia.cloud to the cluster * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the tools cluster === 2024-02-20 === * 16:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 16:12 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-103.tools.eqiad1.wikimedia.cloud to the cluster * 16:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node tools-k8s-worker-102 * 16:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.drain for node tools-k8s-worker-102 * 16:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-101 * 15:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-101 * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:48 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-102 * 15:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-102 * 15:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster * 15:38 taavi@cloudcumin1001: Added a new k8s worker tools-k8s-worker-102.tools.eqiad1.wikimedia.cloud to the cluster * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-56.tools.eqiad1.wikimedia.cloud to the cluster * 12:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-100 * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-100 * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:40 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-55.tools.eqiad1.wikimedia.cloud to the cluster * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-99 * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:29 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-54.tools.eqiad1.wikimedia.cloud to the cluster * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-98 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-98 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-53.tools.eqiad1.wikimedia.cloud to the cluster * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-97 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-97 * 11:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-52.tools.eqiad1.wikimedia.cloud to the cluster * 11:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-96 * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-96 * 11:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-51.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:26 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-50.tools.eqiad1.wikimedia.cloud to the cluster * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:16 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-49.tools.eqiad1.wikimedia.cloud to the cluster * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-95 * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-95 * 10:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-94 * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-93 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-48.tools.eqiad1.wikimedia.cloud to the cluster * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-92 * 10:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-92 * 09:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-6 * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-9.tools.eqiad1.wikimedia.cloud to the cluster * 09:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-47.tools.eqiad1.wikimedia.cloud to the cluster * 09:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 09:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-91 * 09:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-91 * 09:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:15 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-46.tools.eqiad1.wikimedia.cloud to the cluster * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-90 * 08:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-90 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:57 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-45.tools.eqiad1.wikimedia.cloud to the cluster * 08:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-89 * 08:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-44.tools.eqiad1.wikimedia.cloud to the cluster * 08:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-88 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-88 === 2024-02-19 === * 19:04 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 19:03 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-5 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-5 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-43.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-87 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-87 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-42.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-86 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-41.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T357901|T357901]]) * 12:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud * 12:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-85 * 12:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-85 * 12:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:18 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-40.tools.eqiad1.wikimedia.cloud to the cluster * 12:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-84 * 12:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-84 * 12:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:04 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-39.tools.eqiad1.wikimedia.cloud to the cluster * 11:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-83 * 11:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-83 * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:50 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-38.tools.eqiad1.wikimedia.cloud to the cluster * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-82 * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:39 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-37.tools.eqiad1.wikimedia.cloud to the cluster * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-81 * 11:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-81 * 09:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-16 === * 15:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 12:21 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-8.tools.eqiad1.wikimedia.cloud to the cluster * 12:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 10:37 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 10:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:32 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 10:31 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 10:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:59 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-36.tools.eqiad1.wikimedia.cloud to the cluster * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80 * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80 * 09:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:45 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79 * 09:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79 * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78 * 09:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77 * 08:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77 === 2024-02-15 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 * 13:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:02 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster * 12:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 * 12:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 * 12:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:44 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 * 11:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster * 11:37 taavi@cloudcumin1001: Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster * 11:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 * 11:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster === 2024-02-14 === * 19:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30 * 16:35 taavi: kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3 * 16:30 taavi: reboot tools-sgeexec-10-23 due to high load * 09:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud * 09:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:07 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74 * 08:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74 * 08:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:54 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-29.tools.eqiad1.wikimedia.cloud to the cluster * 08:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-73 * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-28.tools.eqiad1.wikimedia.cloud to the cluster * 08:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-72 * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-27.tools.eqiad1.wikimedia.cloud to the cluster * 08:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-71 * 08:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-71 * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-26.tools.eqiad1.wikimedia.cloud to the cluster * 08:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-70 * 08:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-70 * 08:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 08:05 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud to the cluster * 07:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-69 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-69 * 07:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 07:53 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-24.tools.eqiad1.wikimedia.cloud to the cluster * 07:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 07:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-68 * 07:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-68 === 2024-02-13 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-67 * 15:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-23.tools.eqiad1.wikimedia.cloud to the cluster * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-66 * 15:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 15:30 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-22.tools.eqiad1.wikimedia.cloud to the cluster * 15:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 15:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-65 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-65 * 09:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:36 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-21.tools.eqiad1.wikimedia.cloud to the cluster * 09:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-64 * 09:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-64 === 2024-02-12 === * 14:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:58 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-20.tools.eqiad1.wikimedia.cloud to the cluster * 14:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-62 * 14:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 14:47 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-19.tools.eqiad1.wikimedia.cloud to the cluster * 14:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 14:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-61 * 14:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-61 * 13:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-60 * 13:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-60 * 13:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:43 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-18.tools.eqiad1.wikimedia.cloud to the cluster * 13:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-59 * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-58 * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-58 * 13:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:22 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-17.tools.eqiad1.wikimedia.cloud to the cluster * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-57 * 13:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-56 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 13:09 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-16.tools.eqiad1.wikimedia.cloud to the cluster * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-55 * 12:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-54 * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-54 * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15 * 12:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15 * 12:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53 * 12:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52 * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52 * 10:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-11 === * 11:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-02-09 === * 18:03 andrewbogott: updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule * 13:06 taavi: reboot tools-sgecron-2 due to high load * 10:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config * 09:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:56 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster * 09:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 * 09:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 * 08:56 dcaro: restart tools-k8s-worker-50 due to D some stuck processes === 2024-02-08 === * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:46 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster * 09:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 * 09:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 * 09:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 * 09:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:32 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 * 09:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 * 09:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:21 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 09:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 * 09:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 09:10 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster * 09:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:59 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 08:58 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 * 08:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 * 08:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42 === 2024-02-07 === * 21:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers * 18:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 * 17:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 * 17:24 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 17:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers * 17:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers * 17:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers * 16:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all workers === 2024-02-06 === * 13:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 11:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-01-31 === * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-30 === * 19:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:24 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-9.tools.eqiad1.wikimedia.cloud to the cluster * 19:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-9 * 19:16 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 19:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 19:12 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster * 19:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 19:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 19:03 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:51 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 * 18:47 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 * 18:46 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 18:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 18:41 taavi@cloudcumin1001: Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster * 18:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 18:29 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41 * 18:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41 * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40 * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40 * 18:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39 * 18:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39 * 18:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38 * 18:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38 * 18:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37 * 18:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37 * 15:16 dcaro: restart harbor now that the db is clean ([[phab:T356037|T356037]]) * 15:14 dcaro: restart harbor now that the db is clean ([[phab:T3543|T3543]]) * 13:08 taavi: create no-op DMARC record [[phab:T354112|T354112]] * 12:39 dcaro: rebuilding all the toolforge images ([[phab:T354320|T354320]]) * 10:16 dcaro: restarting harbor and flushing redis to regenerate cache data ([[phab:T356037|T356037]]) * 09:33 dcaro: cleaning up old schedules on harbor ([[phab:T356037|T356037]]) === 2024-01-29 === * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36 * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 * 14:36 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud * 14:34 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud * 12:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 12:06 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 11:51 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:37 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster * 11:26 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:23 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 11:22 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster * 11:12 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:12 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35 * 11:10 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-34 * 11:09 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-33 * 11:07 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-33 * 11:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-32 * 11:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-32 * 11:01 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-31 * 10:59 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-30 * 10:57 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster * 10:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster * 10:51 wmbot~taavi@runko: Added a new k8s worker-nfs tools-k8s-worker-nfs-3.tools.eqiad1.wikimedia.cloud to the cluster * 10:46 blancadesal: increased harbor quota for wd-shex-infer to 2GiB * 10:44 blancadesal: increased harbor quota for lucaswerkmeister-test to 2GiB * 10:31 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-26 === * 10:56 taavi: copy helmfile_0.144.0-1_all to bookworm-tools, bookworm-toolsbeta === 2024-01-25 === * 13:17 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 09:54 dcaro: deploy toolforge-jobs-framework-cli 16.0.1 === 2024-01-23 === * 19:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:31 taavi: rebooting tools-sgeexec-10-21, tools-sgeexec-10-22 * 12:58 dcaro: deployed toolforge-envvars-cli 0.0.4 * 10:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-18 === * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17 === 2024-01-17 === * 18:16 dhinus: increase volume quotas for toolsdb [[phab:T344717|T344717]] * 18:14 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) ([[phab:T344717|T344717]]) * 18:14 fnegri@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T344717|T344717]]) * 14:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:56 taavi: update all pre-built docker images [[phab:T352886|T352886]] === 2024-01-15 === * 09:18 taavi: reboot stuck tools-k8s-worker-84 === 2024-01-12 === * 09:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.apt.copy_to_main_repo (exit_code=0) for package 'toolforge-builds-cli' version '0.0.12' * 09:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.apt.copy_to_main_repo for package 'toolforge-builds-cli' version '0.0.12' === 2024-01-11 === * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:14 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:13 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-10 === * 22:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 22:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:17 taavi: reboot tools-k8s-worker-98 === 2024-01-09 === * 23:37 andrewbogott: restarting harbor-db in an attempt to reform harbor -- [[phab:T354714|T354714]] * 23:30 andrewbogott: rebooting tools-harbor-1 in a feeble attempt to get it to work (docker-compose can't restart it) * 23:12 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 23:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 23:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds.builder * 23:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds.builder * 17:31 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:30 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:13 taavi: reboot tools-sgeexec-10-17 due to high load === 2024-01-08 === * 12:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-27, tools-sgeweblight-10-28 * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:17 taavi: reboot tools-sgeexec-10-21 === 2024-01-05 === * 14:55 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:56 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:55 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:29 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 10:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2024-01-04 === * 10:11 dcaro: deploy toolforge-envvars-cli 0.0.3 === 2024-01-03 === * 21:22 andrewbogott: truncating 200 logfiles to 5M on tools nfs * 21:17 andrewbogott: deleting many stray core dumps throughout nfs storage === 2024-01-02 === * 11:06 dcaro: restart toolsdb database to flush connections ([[phab:T354176|T354176]]) * 10:42 dcaro: flushed the redis db on tools-harbor-1 ([[phab:T354176|T354176]]) * 10:37 dcaro: hard reboot tools-harbor-1 * 10:13 dhinus: hard reboot tools-harbor-1 === 2024-01-01 === * 15:55 andrewbogott: rebooting tools-harbor-1, [[phab:T354151|T354151]] ==Archives== * [[Nova Resource:Tools/SAL/Archive 1|Archive 1]] (2013-2014) * [[Nova Resource:Tools/SAL/Archive 2|Archive 2]] (2015-2017) * [[Nova Resource:Tools/SAL/Archive 3|Archive 3]] (2018-2019) * [[Nova Resource:Tools/SAL/Archive 4|Archive 4]] (2020-2021) * [[Nova Resource:Tools/SAL/Archive 5|Archive 5]] (2022-2023) </noinclude> {{SAL|Project Name=tools}} <noinclude>[[Category:SAL]]</noinclude> 1icubxlgq2y2sf6lprfzf2vm7tkabzv 0 4108 2452793 2452774 2026-08-31T07:11:37Z KartikMistry 1115 Removed AG patch. 2452793 wikitext text/x-wiki {{Navigation MediaWiki deployment}} This page tracks '''upcoming''' '''deployments''' of software to the [[:m:Special:SiteMatrix|Wikimedia Foundation servers]]. == Getting started == Ensure you joined the {{irc|wikimedia-operations}} IRC channel as all deployment-related communications happen there. If you need help, contact [[:mw:Wikimedia Release Engineering Team|Release Engineering]] on IRC at {{irc|wikimedia-releng}}; and ping Tyler (<code>thcipriani</code>). * '''MediaWiki is deployed weekly''' through the [[/Train|Deployment Train]]. Other services follow their own schedule. * '''Times are pinned to San Francisco''', thus the UTC time changes in March and November per [[:en:Daylight saving time in the United States|DST]]. * '''Prefer regular [[Backport windows]]''' over adding new windows. To request deployment of a config change or backport, add your username and Gerrit URL to one of the backport windows on this page. You must be online in #wikimedia-operations on IRC during your deployment and install [[WikimediaDebug]] ahead of time. The #wikimedia-operations channel requires you to [[:m:IRC/Instructions#Register your nickname, identify, and enforce|register your nickname]] before you can join. ** You can use the '''backport scheduling tool''' to more easily edit this page: <div style="text-align: center; margin: 1em 0">{{Clickable button 2|:toollabs:schedule-deployment|Schedule a backport|class=mw-ui-progressive}}</div> * Tasks that meet [[/Inclusion criteria|Inclusion criteria]] '''require their own windows''', which includes long-running tasks. '''Schedule more time''' than you think you need to account for delays and set backs, we recommend one hour for most tasks. **To create or modify a recurring deploy window, send a patchset to [[:gitlab:repos/releng/release/-/blob/main/make-deployment-calendar/deployments-calendar.yaml|deployments-calendar.yaml file]] in <code>repos/releng/release.git</code>. **To create an one-off window, simply edit this page accordingly ** '''Announce''' changes to the [[mail:ops|ops mailing list]] ahead of time if you anticipate or are uncertain about noticeable impacts to database load, HTTP caching, or the introduction of new cookies. ** '''Announce''' deployments of major features to the community via [[:m:Tech/News/Next|Tech News]] and/or via other [[:mw:Wikimedia_Product_Guidance/Communication_channels|Product communication channels]]. * '''Something went wrong?''' See [[Incident response]]. Is there a user-impacting problem? Communicate in the {{irc|wikimedia-operations}} IRC channel. If there is a Phabricator task, ensure [[:phab:tag/wikimedia-incident/|#Wikimedia-Incident]] is tagged, and consider setting the [[:mw:Phabricator/Project_management#Priority_levels|Unbreak Now]] priority. __TOC__ {{anchor|Next Week|Near Term|Near term|Near-term}}{{clear}} [[Category:Deployment]] {{Note|content=Subscribe in Google Calendar via <code>wikimedia.org_rudis09ii2mm5fk4hgdjeh1u64@group.calendar.google.com</code>.<br>This may not include one-off windows. '''If there are differences, then the wiki page is canonical and correct'''.}} ==Week of August 31== ==={{Deployment_day|date=2026-08-30}}=== {{Deployment calendar event card |when=2026-08-30 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} ==={{Deployment_day|date=2026-08-31}}=== {{Deployment calendar event card |when=2026-08-31 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|Peterxy|Peterxy}} {{deploy|type=config|gerrit=1323180|title=switch testwiki to use parsoid|status=}} - {{phabricator|T431636}} {{ircnick|hamishcz|Hamish}} {{deploy|type=config|gerrit=1332181|title=thwikibooks: update wordmark and tagline|status=}} - {{phabricator|T436426}} {{ircnick|revi|revi}} {{deploy|type=config|gerrit=1332474|title=kowiki: convert extendedconfirmed calculation to begin from first edit|status=}} - {{phabricator|T436449}} {{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-08-31 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-08-31 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-08-31 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-08-31 08:30 SF |length=0.5 |window=Wikimedia Portals Update |who={{ircnick|jan_drewniak|Jan Drewniak}} |what=Weekly window for the portals page: https://www.wikipedia.org/ }} {{Deployment calendar event card |when=2026-08-31 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-08-31 10:00 SF |length=0.5 |window=Wikidata Query Service weekly deploy |who={{ircnick|ryankemper|Ryan}} |what=... }} {{Deployment calendar event card |when=2026-08-31 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-08-31 14:00 SF |length=2 |window=Weekly Security deployment window |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}} |what=Held deployment window for Security-team related deploys. }} {{Deployment calendar event card |when=2026-08-31 16:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-08-31 19:00 SF |length=1 |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]] |who=N/A |what=Branch <code>wmf/1.47.0-wmf.18</code> }} {{Deployment calendar event card |when=2026-08-31 20:00 SF |length=1 |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]] |who=N/A |what=Deploy <code>wmf/1.47.0-wmf.18</code> to testwikis }} {{Deployment calendar event card |when=2026-08-31 21:00 SF |length=1 |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) |who=N/A |what=Runs <code>scap clean auto</code> }} {{Deployment calendar event card |when=2026-08-31 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-08-31 23:00 SF |length=0.5 |window=Primary database switchover |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}} |what=Held deployment window for database primary masters maintenance }} ==={{Deployment_day|date=2026-09-01}}=== {{Deployment calendar event card |when=2026-09-01 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-01 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-01 05:00 SF |length=1 |window=Mobileapps/RESTBase/Wikifeeds |who=Content Transform Team |what=Content transform team node services (mobileapps/wikifeeds) }} {{Deployment calendar event card |when=2026-09-01 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-01 07:00 SF |length=0.5 |window=Test Kitchen UI Deployment Window |who=Experimentation Platform Team |what=Deployment of Test Kitchen UI (fka MPIC) }} {{Deployment calendar event card |when=2026-09-01 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-01 08:00 SF |length=1 |window=SRE Collaboration Services office hours |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}} |what=Services including Gerrit, Phorge (Phabricator), GitLab }} {{Deployment calendar event card |when=2026-09-01 09:00 SF |length=1 |window=[[Puppet request window]]<br/><small>'''(Max 6 patches)'''</small> |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to Puppet change'' }} {{Deployment calendar event card |when=2026-09-01 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-01 11:00 SF |length=2 |window=MediaWiki train - Utc-7+Utc-0 Version |who={{ircnick|dancy|Ahmon}}, {{ircnick|hashar|Antoine}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.17->1.47.0-wmf.18|1.47.0-wmf.17|1.47.0-wmf.17}} * group0 to [[mw:MediaWiki_1.47/wmf.18|1.47.0-wmf.18]] * '''Blockers: {{phabricator|T430837}}''' }} {{Deployment calendar event card |when=2026-09-01 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-01 14:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-01 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} ==={{Deployment_day|date=2026-09-02}}=== {{Deployment calendar event card |when=2026-09-02 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-02 01:00 SF |length=2 |window=MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) |who={{ircnick|dancy|Ahmon}}, {{ircnick|hashar|Antoine}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.18|1.47.0-wmf.17->1.47.0-wmf.18|1.47.0-wmf.17}} * group1 to [[mw:MediaWiki_1.47/wmf.18|1.47.0-wmf.18]] * '''Blockers: {{phabricator|T430837}}''' }} {{Deployment calendar event card |when=2026-09-02 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-02 04:00 SF |length=1 |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]] |who=Marielle ({{ircnick|mvolz}}) |what=See [[mw:Citoid|Citoid]] }} {{Deployment calendar event card |when=2026-09-02 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-02 07:00 SF |length=1 |window=Wikifunctions Services UTC Afternoon |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas) |what=Wikifunctions back-end k8s services }} {{Deployment calendar event card |when=2026-09-02 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-02 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-02 11:00 SF |length=2 |window=MediaWiki train - Utc-7+Utc-0 Version |who={{ircnick|dancy|Ahmon}}, {{ircnick|hashar|Antoine}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.18|1.47.0-wmf.17->1.47.0-wmf.18|1.47.0-wmf.17}} * group1 to [[mw:MediaWiki_1.47/wmf.18|1.47.0-wmf.18]] * '''Blockers: {{phabricator|T430837}}''' }} {{Deployment calendar event card |when=2026-09-02 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-02 14:00 SF |length=1 |window=Wikifunctions Services UTC Late |who=Abstract Wikipedia team (North and South America) |what=Wikifunctions back-end k8s services }} {{Deployment calendar event card |when=2026-09-02 15:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-02 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-02 23:00 SF |length=0.5 |window=Primary database switchover |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}} |what=Held deployment window for database primary masters maintenance }} ==={{Deployment_day|date=2026-09-03}}=== {{Deployment calendar event card |when=2026-09-03 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-03 01:00 SF |length=2 |window=MediaWiki train - Utc-7+Utc-0 Version (secondary timeslot) |who={{ircnick|dancy|Ahmon}}, {{ircnick|hashar|Antoine}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.18|1.47.0-wmf.18|1.47.0-wmf.17->1.47.0-wmf.18}} * group2 to [[mw:MediaWiki_1.47/wmf.18|1.47.0-wmf.18]] * '''Blockers: {{phabricator|T430837}}''' }} {{Deployment calendar event card |when=2026-09-03 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-03 05:00 SF |length=1 |window=Mobileapps/RESTBase/Wikifeeds |who=Content Transform Team |what=Content transform team node services (mobileapps/wikifeeds) }} {{Deployment calendar event card |when=2026-09-03 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-03 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-03 09:00 SF |length=1 |window=[[Puppet request window]]<br/><small>'''(Max 6 patches)'''</small> |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to Puppet change'' }} {{Deployment calendar event card |when=2026-09-03 10:00 SF |length=1 |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) |who={{ircnick|bd808}} |what=... }} {{Deployment calendar event card |when=2026-09-03 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-03 11:00 SF |length=2 |window=MediaWiki train - Utc-7+Utc-0 Version |who={{ircnick|dancy|Ahmon}}, {{ircnick|hashar|Antoine}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.18|1.47.0-wmf.18|1.47.0-wmf.17->1.47.0-wmf.18}} * group2 to [[mw:MediaWiki_1.47/wmf.18|1.47.0-wmf.18]] * '''Blockers: {{phabricator|T430837}}''' }} {{Deployment calendar event card |when=2026-09-03 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-03 14:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-03 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} ==={{Deployment_day|date=2026-09-04}}=== {{Deployment calendar event card |when=2026-09-04 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} {{Deployment calendar event card |when=2026-09-04 04:00 SF |length=0.5 |window=GitLab version upgrades |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}} |what=GitLab version upgrades }} ==={{Deployment_day|date=2026-09-05}}=== {{Deployment calendar event card |when=2026-09-05 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} ==Week of September 07== ==={{Deployment_day|date=2026-09-06}}=== {{Deployment calendar event card |when=2026-09-06 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} ==={{Deployment_day|date=2026-09-07}}=== {{Deployment calendar event card |when=2026-09-07 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-07 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-07 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-07 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-07 08:30 SF |length=0.5 |window=Wikimedia Portals Update |who={{ircnick|jan_drewniak|Jan Drewniak}} |what=Weekly window for the portals page: https://www.wikipedia.org/ }} {{Deployment calendar event card |when=2026-09-07 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-07 10:00 SF |length=0.5 |window=Wikidata Query Service weekly deploy |who={{ircnick|ryankemper|Ryan}} |what=... }} {{Deployment calendar event card |when=2026-09-07 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-07 14:00 SF |length=2 |window=Weekly Security deployment window |who={{ircnick|alexsanford|Alex}}, {{ircnick|Reedy|Sam}}, {{ircnick|sbassett|Scott}}, {{ircnick|Maryum|Maryum}}, {{ircnick|manfredi|Manfredi}} |what=Held deployment window for Security-team related deploys. }} {{Deployment calendar event card |when=2026-09-07 16:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-07 19:00 SF |length=1 |window=Automatic branching of MediaWiki, extensions, skins, and vendor – see [[Heterogeneous deployment/Train deploys]] |who=N/A |what=Branch <code>wmf/1.47.0-wmf.19</code> }} {{Deployment calendar event card |when=2026-09-07 20:00 SF |length=1 |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]] |who=N/A |what=Deploy <code>wmf/next</code> to pretrain environment }} {{Deployment calendar event card |when=2026-09-07 20:00 SF |length=1 |window=Automatic deployment of MediaWiki, extensions, skins, and vendor to testwikis only – see [[Heterogeneous deployment/Train deploys]] |who=N/A |what=Deploy <code>wmf/1.47.0-wmf.19</code> to testwikis }} {{Deployment calendar event card |when=2026-09-07 21:00 SF |length=1 |window=Automatic removal of all obsolete MediaWiki versions from the deployment and bare metal servers (except the most-recent obsolete version) |who=N/A |what=Runs <code>scap clean auto</code> }} {{Deployment calendar event card |when=2026-09-07 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-07 23:00 SF |length=0.5 |window=Primary database switchover |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}} |what=Held deployment window for database primary masters maintenance }} ==={{Deployment_day|date=2026-09-08}}=== {{Deployment calendar event card |when=2026-09-08 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-08 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-08 05:00 SF |length=1 |window=Mobileapps/RESTBase/Wikifeeds |who=Content Transform Team |what=Content transform team node services (mobileapps/wikifeeds) }} {{Deployment calendar event card |when=2026-09-08 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-08 07:00 SF |length=0.5 |window=Test Kitchen UI Deployment Window |who=Experimentation Platform Team |what=Deployment of Test Kitchen UI (fka MPIC) }} {{Deployment calendar event card |when=2026-09-08 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-08 08:00 SF |length=1 |window=SRE Collaboration Services office hours |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}} |what=Services including Gerrit, Phorge (Phabricator), GitLab }} {{Deployment calendar event card |when=2026-09-08 09:00 SF |length=1 |window=[[Puppet request window]]<br/><small>'''(Max 6 patches)'''</small> |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to Puppet change'' }} {{Deployment calendar event card |when=2026-09-08 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-08 11:00 SF |length=2 |window=MediaWiki train - Utc-7 Version |who={{ircnick|dduvall|Dan}}, {{ircnick|dancy|Ahmon}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.18->1.47.0-wmf.19|1.47.0-wmf.18|1.47.0-wmf.18}} * group0 to [[mw:MediaWiki_1.47/wmf.19|1.47.0-wmf.19]] * '''Blockers: {{phabricator|T430838}}''' }} {{Deployment calendar event card |when=2026-09-08 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-08 14:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-08 20:00 SF |length=1 |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]] |who=N/A |what=Deploy <code>wmf/next</code> to pretrain environment }} {{Deployment calendar event card |when=2026-09-08 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} ==={{Deployment_day|date=2026-09-09}}=== {{Deployment calendar event card |when=2026-09-09 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-09 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-09 04:00 SF |length=1 |window=[[mw:Services|Services]] – [[Citoid]] / [[Zotero]] |who=Marielle ({{ircnick|mvolz}}) |what=See [[mw:Citoid|Citoid]] }} {{Deployment calendar event card |when=2026-09-09 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-09 07:00 SF |length=1 |window=Wikifunctions Services UTC Afternoon |who=Abstract Wikipedia team (Africa, Europe, Eastern Americas) |what=Wikifunctions back-end k8s services }} {{Deployment calendar event card |when=2026-09-09 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-09 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-09 11:00 SF |length=2 |window=MediaWiki train - Utc-7 Version |who={{ircnick|dduvall|Dan}}, {{ircnick|dancy|Ahmon}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.19|1.47.0-wmf.18->1.47.0-wmf.19|1.47.0-wmf.18}} * group1 to [[mw:MediaWiki_1.47/wmf.19|1.47.0-wmf.19]] * '''Blockers: {{phabricator|T430838}}''' }} {{Deployment calendar event card |when=2026-09-09 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-09 14:00 SF |length=1 |window=Wikifunctions Services UTC Late |who=Abstract Wikipedia team (North and South America) |what=Wikifunctions back-end k8s services }} {{Deployment calendar event card |when=2026-09-09 15:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-09 20:00 SF |length=1 |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]] |who=N/A |what=Deploy <code>wmf/next</code> to pretrain environment }} {{Deployment calendar event card |when=2026-09-09 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-09 23:00 SF |length=0.5 |window=Primary database switchover |who={{ircnick|marostegui|Manuel Arostegui}}, {{ircnick|cezmunsta|Ceri Williams}}, {{ircnick|federico3|Federico Ceratto}} |what=Held deployment window for database primary masters maintenance }} ==={{Deployment_day|date=2026-09-10}}=== {{Deployment calendar event card |when=2026-09-10 00:00 SF |length=1 |window=[[Backport windows|UTC morning backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Amir1|Amir}}, {{ircnick|urbanecm|Martin}}, {{ircnick|awight|Adam}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-10 03:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC mid-day) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-10 05:00 SF |length=1 |window=Mobileapps/RESTBase/Wikifeeds |who=Content Transform Team |what=Content transform team node services (mobileapps/wikifeeds) }} {{Deployment calendar event card |when=2026-09-10 06:00 SF |length=1 |window=[[Backport windows|UTC afternoon backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|Lucas_WMDE|Lucas}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-10 07:30 SF |length=0.5 |window=Test Kitchen Experiment Deployment Window |who=Test Kitchen |what=Automatic start/stop of active experiments and instruments managed by [[Test Kitchen]]. }} {{Deployment calendar event card |when=2026-09-10 09:00 SF |length=1 |window=[[Puppet request window]]<br/><small>'''(Max 6 patches)'''</small> |who={{ircnick|jhathaway|JHathaway}}, {{ircnick|rzl|Reuven}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to Puppet change'' }} {{Deployment calendar event card |when=2026-09-10 10:00 SF |length=1 |window=Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker) |who={{ircnick|bd808}} |what=... }} {{Deployment calendar event card |when=2026-09-10 10:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC late) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} {{Deployment calendar event card |when=2026-09-10 11:00 SF |length=2 |window=MediaWiki train - Utc-7 Version |who={{ircnick|dduvall|Dan}}, {{ircnick|dancy|Ahmon}} |what=[[mw:MediaWiki 1.47/Roadmap#Schedule for the deployments|1.47 schedule]] {{DeployOneWeekMini|1.47.0-wmf.19|1.47.0-wmf.19|1.47.0-wmf.18->1.47.0-wmf.19}} * group2 to [[mw:MediaWiki_1.47/wmf.19|1.47.0-wmf.19]] * '''Blockers: {{phabricator|T430838}}''' }} {{Deployment calendar event card |when=2026-09-10 13:00 SF |length=1 |window=[[Backport windows|UTC late backport window]]<br/><small>'''Your patch may or may not be deployed at the sole discretion of the deployer'''</small> |who={{ircnick|RoanKattouw|Roan}}, {{ircnick|urbanecm|Martin}}, {{ircnick|TheresNoTime|Sammy}}, {{ircnick|kindrobot|Stef}}, {{ircnick|cjming|Clare}} |what={{ircnick|irc-nickname|Requesting Developer}} * ''Gerrit link to backport or config change'' }} {{Deployment calendar event card |when=2026-09-10 14:00 SF |length=1 |window=Readers deployment window |who=Readers |what=NOTE: often skipped, the reader teams do not typically check IRC so assume this is not being used if 5 minutes past the start }} {{Deployment calendar event card |when=2026-09-10 20:00 SF |length=1 |window=Automatic deployment of MediaWiki to pretrain wikis - see [[mw:Pretrain]] |who=N/A |what=Deploy <code>wmf/next</code> to pretrain environment }} {{Deployment calendar event card |when=2026-09-10 23:00 SF |length=1 |window=[[MediaWiki_On_Kubernetes#How_to_manage_changes_to_the_infrastructure|MediaWiki infrastructure]] (UTC early) |who=SRE team |what=MediaWiki-related infrastructure changes that need a kubernetes deployment. }} ==={{Deployment_day|date=2026-09-11}}=== {{Deployment calendar event card |when=2026-09-11 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} {{Deployment calendar event card |when=2026-09-11 04:00 SF |length=0.5 |window=GitLab version upgrades |who={{ircnick|jelto|Jelto}}, {{ircnick|arnoldokoth|Arnold}}, {{ircnick|mutante|Daniel}}, {{ircnick|arnaudb|Arnaud}} |what=GitLab version upgrades }} ==={{Deployment_day|date=2026-09-12}}=== {{Deployment calendar event card |when=2026-09-12 00:00 SF |length=24 |window=No deploys all day! See [[Deployments/Emergencies]] if things are broken. |who= |what=No Deploys }} jaxyf27z2f1oaa6c4uxk70zk04c2fo6 498 6726 2452776 2452282 2026-08-30T20:12:34Z Stashbot 7414 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api 2452776 wikitext text/x-wiki === 2026-08-30 === * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 10:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-08-26 === * 09:49 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:14 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:00 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:59 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:54 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 08:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-20 === * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:06 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:33 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:52 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 11:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:38 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:38 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2026-08-13 === * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 02:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 01:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13 * 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13 * 01:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13 * 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) === 2026-08-12 === * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 19:05 andrew@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 18:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:02 andrewbogott: upgrading to kubernetes 1.33.13 * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 14:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-05 === * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 14:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-04 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:51 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 09:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-03 === * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-harbor-1 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-harbor-1 * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:25 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-31 === * 12:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:53 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:44 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2026-07-27 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:14 godog: set 'dumps_use_nfs_lb: true' for prefix-puppet toolsbeta-test-k8s-worker-nfs - [[phab:T432325|T432325]] * 10:30 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T432325|T432325]]) * 10:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 09:32 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 09:29 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) === 2026-07-23 === * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-20 === * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:29 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:20 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:19 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-16 === * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:49 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:07 godog: fail back to classic dumps nfs for toolsbeta-test-k8s-worker-nfs prefix * 12:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-harbor' * 12:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-harbor' * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-5 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-5 * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-4 * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-4 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-cumin-1 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-cumin-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-cumin' * 10:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-cumin' * 08:49 godog: set 'dumps_use_nfs_lb: true' for mail, bastion, static prefixes * 08:22 godog: test reboot all workers * 07:58 godog: test dumps-nfs switchover on toolsbeta-test-k8s-worker-nfs-8 * 07:44 godog: set 'dumps_use_nfs_lb: true' for toolsbeta-test-k8s-worker-nfs prefix * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 21:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:25 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud === 2026-07-09 === * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 09:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 09:35 godog: carry out nfs tests on toolsbeta-test-k8s-worker-nfs-8 - [[phab:T411248|T411248]] * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 16:37 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 16:28 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 16:28 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-06-24 === * 16:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:30 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-ap * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-ap === 2026-06-15 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-06-11 === * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-08 === * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:03 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:14 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) === 2026-06-04 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-06-02 === * 15:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 01:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 19:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 07:54 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 07:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-05-21 === * 09:48 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:33 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:29 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:13 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2026-05-20 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:27 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:26 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 00:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 23:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 23:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 09:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 09:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-14 === * 15:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway ([[phab:T425479|T425479]]) * 10:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway ([[phab:T425479|T425479]]) === 2026-05-13 === * 17:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:19 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:36 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-11 === * 17:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:23 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 15:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:35 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-06 === * 08:57 taavi: renew prometheus kubernetes certificate about to expire * 08:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-05-05 === * 04:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 19:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 10:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node === 2026-04-21 === * 14:10 taavi: delete all ingress objects from toolsbeta [[phab:T392356|T392356]] * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-10 === * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-04-07 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 13:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 07:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 07:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 17:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:20 godog: reboot harbordb1 instance * 12:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-26 === * 17:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-24 === * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 14:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-03-17 === * 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2026-03-10 === * 18:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 11:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice-cli * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice-cli * 15:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:53 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 65536 ram * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 65536 ram * 12:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-gateway-1 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-gateway-1 * 11:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster === 2026-03-06 === * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-03-05 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:02 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 14:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:11 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-04 === * 21:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:30 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:21 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:18 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:34 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:21 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 04:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 04:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 04:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 04:15 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 04:14 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:04 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:31 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:09 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:43 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:45 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:40 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:39 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:37 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:27 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:25 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 00:04 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) === 2026-03-03 === * 23:55 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:54 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:53 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:51 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:49 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 23:01 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:55 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:41 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:40 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:29 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:56 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 21:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:27 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:17 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 20:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:50 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:01 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:19 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-02-25 === * 14:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component gateway-api * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-24 === * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-16 === * 09:10 dcaro: restarted the admin tool, was giving error connecting to lighttpd socket === 2026-02-05 === * 18:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-02-04 === * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-02-03 === * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-01-28 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-12 === * 17:59 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 02:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:13 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 01:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 00:56 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 00:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 100k * 11:06 godog: bump object quota to 200G === 2025-12-15 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics === 2025-12-11 === * 16:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:41 andrewbogott: deleting toolsbeta-test-k8s-etcd-27 and replacing with a Bullseye node for cluster consistency [[phab:T361237|T361237]] === 2025-12-03 === * 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) === 2025-12-02 === * 22:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster === 2025-12-01 === * 16:22 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-25 === * 13:27 volans: temporarily disabling puppet on the nfs workers to merge gerrit/1210582 * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-prometheus-2 (cluster eqiad1) * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-prometheus-2 (cluster eqiad1) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:03 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 14:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:06 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:27 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:27 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-19 === * 16:50 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:33 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:18 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:11 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:11 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:10 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:46 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-18 === * 17:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:59 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-11-17 === * 18:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 18:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-13 === * 15:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-bastion-7 (cluster eqiad1) * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-bastion-7 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) === 2025-11-12 === * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-10 === * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 20:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-06 === * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-04 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 00:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 20:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-11-01 === * 00:06 bd808: Re-enabled Puppet on toolsbeta-test-k8s-haproxy-7 ([[phab:T408570|T408570]]) === 2025-10-31 === * 19:53 bd808: Disabled Puppet on toolsbeta-test-k8s-haproxy-7 for manual testing of haproxy changes ([[phab:T408570|T408570]]) === 2025-10-30 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-10-27 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:10 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-23 === * 17:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:08 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-10-21 === * 11:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 00:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-16 === * 11:20 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:12 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-14 === * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-13 === * 14:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-08 === * 11:40 godog: nfs back up and has been for some time [[phab:T406688|T406688]] * 10:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:19 godog: shut down nfs while investigating [[phab:T406688|T406688]] * 09:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-07 === * 18:56 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:56 root@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:56 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 11:52 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:46 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 11:44 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:56 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 08:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 08:39 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:39 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:37 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:36 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:35 godog: move nfs from toolsbeta-nfs-4 to toolsbeta-nfs-5 - [[phab:T404584|T404584]] * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-10-03 === * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-6 * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-5 * 09:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-5 === 2025-10-02 === * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) === 2025-10-01 === * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-29 === * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found === 2025-09-26 === * 08:17 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 08:07 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-25 === * 17:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:39 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 godog: test reboot of toolsbeta-nfs-4 * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 17:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:31 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:19 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:17 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 09:11 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 09:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:03 filippo@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-14 * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 08:42 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-bastion-7 * 08:41 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-bastion-7 * 08:19 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:19 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:03 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 07:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 06:34 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 06:26 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-23 === * 15:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-09-22 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-18 === * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-09-16 === * 16:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:56 taavi: delete toolsbeta-sgebastion puppet prefix * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:43 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:42 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 15:35 wmbot~dcaro@acme: Added a new k8s ingress toolsbeta-test-k8s-ingress-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:24 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:24 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:24 wmbot~dcaro@acme: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:11 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 14:09 wmbot~dcaro@acme: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 13:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 13:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-5 * 13:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-5 * 13:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:31 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-test-k8s-worker-nfs-5 * 13:20 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-test-k8s-worker-nfs-5 * 13:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-15 === * 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-09-10 === * 14:43 godog: reset puppet to production, testing finished * 14:39 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186938 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186937 * 08:40 volans: add volans to toolsbeta.admin tool * 08:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'volans' in role 'member' * 08:15 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member' === 2025-09-09 === * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component mainain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component mainain-kubeusers === 2025-09-08 === * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2025-09-05 === * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro: bumped the object quota to 100G, like tools === 2025-09-03 === * 17:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 23:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-01 === * 21:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 21:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 21:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:26 dcaro: upgrade finished * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=97) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:59 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:26 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:26 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.run_tests * 08:06 dcaro: starting upgrade to v1.30 ([[phab:T402377|T402377]]) === 2025-08-28 === * 08:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 18:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 17:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:38 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-21 === * 10:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:21 dcaro: test for sal === 2025-08-19 === * 16:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-14 === * 15:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:23 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) , no stuck workers found * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers , no stuck workers found * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 09:50 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 03:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:14 dcaro: deploying kyverno ([[phab:T394787|T394787]]) === 2025-08-08 === * 06:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 05:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:34 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli ([[phab:T401014|T401014]]) * 17:33 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:21 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 17:21 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 17:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2025-08-05 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tool-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tool-webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 08:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 02:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 02:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 01:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 01:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 01:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 00:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) === 2025-08-01 === * 03:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 03:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 15:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 03:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-30 === * 09:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-28 === * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-23 === * 18:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-14 === * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398715|T398715]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398715|T398715]]) === 2025-07-10 === * 10:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-09 === * 17:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:07 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:46 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component misctools-cli * 08:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 08:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2025-07-08 === * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-07-07 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-03 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-02 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-06-26 === * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-24 === * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging * 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-06-23 === * 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-19 === * 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-18 === * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-06-17 === * 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-16 === * 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-12 === * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-11 === * 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-10 === * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]]) * 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]]) === 2025-06-09 === * 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-07 === * 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G === 2025-06-06 === * 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-05 === * 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-06-04 === * 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-02 === * 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-22 === * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 === 2025-05-21 === * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-20 === * 18:24 bd808: Made addshore an admin === 2025-05-19 === * 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-15 === * 08:13 taavi: renew expiring Puppet CA cert === 2025-05-14 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-12 === * 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-05-09 === * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-08 === * 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-07 === * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-24 === * 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-04-23 === * 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-21 === * 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]] === 2025-04-17 === * 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]] === 2025-04-11 === * 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-09 === * 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-07 === * 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-04 === * 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-31 === * 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) === 2025-03-25 === * 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-13 === * 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-12 === * 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder * 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-06 === * 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-05 === * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-04 === * 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2025-03-03 === * 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-02-27 === * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-02-26 === * 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-02-24 === * 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-19 === * 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-17 === * 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-06 === * 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-01 === * 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]] * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes === 2025-01-29 === * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-23 === * 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]]) * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]]) * 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-22 === * 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-21 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10 * 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers * 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10 === 2025-01-20 === * 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-17 === * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-15 === * 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-07 === * 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-06 === * 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2024-12-13 === * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-12-06 === * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-29 === * 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-25 === * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-23 === * 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]]) * 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]]) * 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice * 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice === 2024-11-18 === * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-14 === * 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-07 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-06 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-30 === * 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) === 2024-10-29 === * 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1 * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1 === 2024-10-16 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-10 === * 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-10-09 === * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers === 2024-10-04 === * 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-03 === * 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]] * 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) === 2024-10-01 === * 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-28 === * 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]]) * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]]) * 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10 * 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster * 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-24 === * 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-21 === * 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 === 2024-09-20 === * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-11 === * 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-09-10 === * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) === 2024-09-09 === * 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-06 === * 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-09-05 === * 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8 * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7 * 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7 * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-03 === * 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno * 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-09-02 === * 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-08-28 === * 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back. * 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico === 2024-08-26 === * 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-21 === * 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests * 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-13 === * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-08-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components * 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents === 2024-08-06 === * 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-05 === * 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-30 === * 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli === 2024-07-29 === * 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-18 === * 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-12 === * 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 === 2024-07-11 === * 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]] * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26 * 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26 * 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25 * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25 * 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23 * 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]]) * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-25 === * 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-24 === * 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-21 === * 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd === 2024-06-20 === * 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-17 === * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7 * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10 * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10 * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5 * 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1 * 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5 * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5 * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6 * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6 * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2 * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2 === 2024-06-14 === * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1 * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1 * 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-07 === * 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-05-30 === * 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-29 === * 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-28 === * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-25 === * 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-15 === * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-05-10 === * 13:57 taavi: renew k8s prometheus certificate === 2024-05-07 === * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-04 === * 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json * 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-24 === * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-15 === * 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison * 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2024-04-11 === * 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-10 === * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-04-09 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-08 === * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-05 === * 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-02 === * 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk * 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02 === 2024-04-01 === * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-03-28 === * 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]]) * 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3 * 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy' * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy' * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' === 2024-03-27 === * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2 * 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2 === 2024-03-26 === * 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) * 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2024-03-25 === * 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector * 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector === 2024-03-22 === * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-21 === * 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4 * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4 * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3 * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-19 === * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-03-18 === * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-16 === * 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8 === 2024-03-15 === * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01 * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) === 2024-03-13 === * 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-12 === * 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-11 === * 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-03-07 === * 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-05 === * 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-04 === * 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-28 === * 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud * 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-02-22 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-02-21 === * 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-20 === * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 === 2024-02-19 === * 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-02-15 === * 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-02-13 === * 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4 * 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4 * 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3 * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3 * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7 * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7 === 2024-02-12 === * 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-09 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2024-02-08 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6 * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6 * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10 === 2024-02-06 === * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-02-05 === * 09:55 arturo: grant myself member and admin privileges === 2024-01-31 === * 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-29 === * 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-01-26 === * 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-01-25 === * 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-01-23 === * 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-17 === * 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-12 === * 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1 === 2024-01-11 === * 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-09 === * 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-08 === * 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-05 === * 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-12-26 === * 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable === 2023-12-20 === * 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase === 2023-12-15 === * 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]]) * 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]]) === 2023-12-13 === * 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97) * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec * 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]]) * 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]]) * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]]) * 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]]) * 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]]) * 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-12-12 === * 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) === 2023-12-11 === * 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]]) * 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]]) * 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]]) === 2023-12-07 === * 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-05 === * 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-12-04 === * 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-01 === * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-11-23 === * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-22 === * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-11-20 === * 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-17 === * 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-09 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-01 === * 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99) * 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-30 === * 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-27 === * 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-10-26 === * 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-25 === * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster * 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster === 2023-10-23 === * 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-20 === * 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]]) * 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]]) === 2023-10-18 === * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-13 === * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97) * 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-12 === * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-10 === * 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-09 === * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-05 === * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-04 === * 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-10-03 === * 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2023-09-27 === * 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2023-09-25 === * 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-20 === * 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-19 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-15 === * 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-09-14 === * 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer * 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer * 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission * 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission * 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 10:16 dcaro: deploy bulids-api 0.0.96 * 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-13 === * 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone * 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone === 2023-09-11 === * 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]]) * 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]]) === 2023-09-06 === * 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]]) === 2023-09-05 === * 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) === 2023-08-31 === * 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0) * 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs * 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 === 2023-08-30 === * 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]]) * 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]] === 2023-08-29 === * 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko * 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-28 === * 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay === 2023-08-22 === * 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko === 2023-08-18 === * 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-17 === * 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}} === 2023-08-11 === * 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn === 2023-07-26 === * 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko === 2023-07-25 === * 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko === 2023-07-20 === * 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]]) * 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta === 2023-07-18 === * 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]]) === 2023-07-13 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko === 2023-07-12 === * 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]]) === 2023-07-04 === * 13:55 taavi: removed floating IP and public dns records for the harbor server === 2023-07-03 === * 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko * 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay === 2023-06-26 === * 07:49 dcaro: restarting harbor trove DB (in error status) === 2023-06-21 === * 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]]) * 11:48 dcaro: deploy bulids-api 0.2.0 === 2023-06-16 === * 14:28 dcaro: deployed envvars-api 0.0.1 * 07:41 dcaro: deployed latest builds-api 0.1.0 === 2023-06-15 === * 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye === 2023-06-08 === * 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]]) === 2023-06-07 === * 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko === 2023-06-01 === * 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-26 === * 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-25 === * 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]]) === 2023-05-24 === * 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]]) * 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]]) === 2023-05-23 === * 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169 === 2023-05-16 === * 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]]) * 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus * 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98 === 2023-05-15 === * 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus * 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus === 2023-05-09 === * 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko * 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2023-05-05 === * 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko === 2023-05-01 === * 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu === 2023-04-30 === * 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko * 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko * 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko === 2023-04-19 === * 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko === 2023-04-11 === * 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus * 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo * 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko * 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko * 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko * 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko === 2023-04-05 === * 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko * 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko === 2023-04-03 === * 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo * 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo * 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo * 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo * 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo * 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo * 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo === 2023-03-19 === * 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko === 2023-03-14 === * 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local * 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local === 2023-03-13 === * 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-03-10 === * 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko === 2023-03-09 === * 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko === 2023-03-07 === * 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]] === 2023-03-06 === * 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko * 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko * 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko * 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1 === 2023-03-05 === * 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko === 2023-03-02 === * 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836) === 2023-03-01 === * 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko === 2023-02-28 === * 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko * 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko * 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-02-23 === * 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko * 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko === 2023-02-17 === * 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance * 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance === 2023-02-16 === * 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo * 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko * 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml === 2023-02-15 === * 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo === 2023-02-14 === * 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]] * 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]]) === 2023-02-13 === * 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko * 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo * 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:15 arturo: cordoned & drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]]) * 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo * 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-10 === * 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-01 === * 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko === 2023-01-26 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko === 2023-01-23 === * 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko === 2023-01-20 === * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo === 2023-01-19 === * 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff) === 2023-01-18 === * 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo === 2023-01-17 === * 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance * 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance * 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud === 2023-01-10 === * 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance * 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance === 2022-12-09 === * 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]]) === 2022-11-30 === * 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo === 2022-11-29 === * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko * 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko * 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko === 2022-11-28 === * 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko * 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko === 2022-11-15 === * 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu === 2022-11-14 === * 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 === 2022-11-07 === * 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169 === 2022-11-04 === * 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo === 2022-11-01 === * 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot === 2022-10-24 === * 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye * 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye * 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus === 2022-10-18 === * 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo === 2022-10-17 === * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-10-14 === * 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko === 2022-10-12 === * 10:29 dcaro: deploying new registry-admission controller === 2022-10-10 === * 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko === 2022-09-28 === * 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]]) === 2022-09-27 === * 14:23 arturo: briefly livehacking puppetmaster === 2022-08-24 === * 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko === 2022-08-12 === * 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again === 2022-08-03 === * 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap * 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station === 2022-08-01 === * 14:01 taavi: unbreak acme-chief after keystone communication issues === 2022-07-19 === * 15:45 taavi: deploying and testing maintain-kubeusers updates === 2022-06-28 === * 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko === 2022-06-24 === * 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko === 2022-06-19 === * 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02 === 2022-06-03 === * 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]]) * 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko === 2022-05-30 === * 13:42 taavi: run grid-configurator to remove stale config for some removed nodes === 2022-05-26 === * 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko === 2022-04-20 === * 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko === 2022-04-15 === * 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]] === 2022-04-11 === * 14:47 dcaro: deploying custom version of the regitsry admission hook === 2022-04-08 === * 10:45 arturo: disabled debug mode on the k8s jobs-emailer component === 2022-04-05 === * 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo * 07:21 arturo: deploying toolforge-jobs-framework-cli v7 === 2022-04-04 === * 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo * 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions === 2022-03-25 === * 11:31 dcaro: All alerting VMs rebooted, checking that everything is "working" ([[phab:T304672|T304672]]) * 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]]) * 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]]) * 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]]) === 2022-03-16 === * 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta === 2022-03-15 === * 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo === 2022-03-14 === * 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-03-11 === * 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]]) * 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]]) * 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]]) * 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]]) === 2022-03-08 === * 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates === 2022-03-07 === * 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]]) === 2022-02-17 === * 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access === 2022-02-08 === * 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]]) * 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles * 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]]) === 2022-02-04 === * 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]] === 2022-01-28 === * 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo === 2022-01-25 === * 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2022-01-20 === * 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-19 === * 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta === 2022-01-18 === * 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster * 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 && git cherry-pick FETCH_HEAD * 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix) === 2022-01-17 === * 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo * 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo * 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo === 2022-01-14 === * 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-12 === * 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo * 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig` * 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig' * 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo === 2022-01-11 === * 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2021-12-23 === * 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance * 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance === 2021-12-22 === * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance * 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance * 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance * 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance === 2021-12-21 === * 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance * 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]]) === 2021-12-16 === * 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance === 2021-12-15 === * 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance === 2021-12-08 === * 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028 === 2021-11-28 === * 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance === 2021-11-16 === * 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]] === 2021-11-05 === * 19:07 majavah: testing registry-admission changes === 2021-10-28 === * 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true` === 2021-10-25 === * 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]] === 2021-10-20 === * 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories === 2021-10-16 === * 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]] === 2021-10-14 === * 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus === 2021-10-13 === * 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-12 === * 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master === 2021-10-11 === * 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-07 === * 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-06 === * 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-04 === * 17:07 bstorm: reboot everything [[phab:T291406|T291406]] * 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]] * 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting) * 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]]) === 2021-10-01 === * 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) * 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) === 2021-09-29 === * 22:13 bstorm: ran label fix script to use new label format * 22:12 bstorm: toollabs-webservice 0.77 deployed === 2021-09-28 === * 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api === 2021-09-27 === * 16:13 majavah: testing volume-admission fix for containers with some volumes mounted === 2021-09-23 === * 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]] === 2021-09-22 === * 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a "fair" test battery against [[phab:T291406|T291406]] === 2021-09-15 === * 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]] === 2021-09-14 === * 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]] * 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]]) * 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]]) * 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]]) === 2021-09-13 === * 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]] === 2021-09-09 === * 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]] * 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]] * 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml * 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp === 2021-09-03 === * 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]] === 2021-08-30 === * 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]] === 2021-08-21 === * 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]] === 2021-08-20 === * 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes === 2021-08-12 === * 16:55 bstorm: deployed updated manifest for ingress-admission * 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]] === 2021-07-30 === * 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]] === 2021-07-29 === * 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]] * 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]] === 2021-07-23 === * 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]] === 2021-07-22 === * 15:32 arturo: re-deploying toolforge-jobs-framework-api === 2021-07-21 === * 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]]) * 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-<nowiki>{</nowiki>apiserver,controller-manager<nowiki>}</nowiki>.yaml in all 3 control nodes ([[phab:T286108|T286108]]) * 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]]) * 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848 === 2021-07-20 === * 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]] * 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]] * 08:36 majavah: resolve merge conflicts on labs/private === 2021-07-16 === * 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]] * 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]]) === 2021-07-15 === * 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}} === 2021-07-14 === * 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]] === 2021-07-12 === * 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues === 2021-07-07 === * 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work === 2021-07-01 === * 17:01 majavah: updating jobs-framework-api * 10:00 arturo: refreshed jobs-api deployment === 2021-06-29 === * 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus === 2021-06-28 === * 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus * 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-25 === * 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-24 === * 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers * 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-22 === * 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18 === 2021-06-17 === * 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again === 2021-06-16 === * 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100 === 2021-06-15 === * 16:10 majavah: set toolsbeta-bastion-05 as grid submit host === 2021-06-14 === * 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]] * 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]]) === 2021-06-12 === * 14:42 majavah: sync hiera key prometheus_nodes to match tools === 2021-06-11 === * 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes * 14:54 majavah: generate and add own root key to passwords::root::extra_keys === 2021-06-08 === * 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]] * 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]] * 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]] * 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work * 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]] === 2021-06-03 === * 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]] * 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]] === 2021-05-25 === * 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 * 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5 * 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020 === 2021-05-24 === * 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes === 2021-05-19 === * 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/ * 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1 === 2021-05-15 === * 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]] === 2021-05-14 === * 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]]) * 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]]) === 2021-05-13 === * 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]] === 2021-05-10 === * 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes * 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]] * 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]] === 2021-05-08 === * 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]] === 2021-05-07 === * 17:00 bstorm: deleted "toolsbeta-test-k8s-haproxy-2", "toolsbeta-test-k8s-haproxy-1" when the dns caches finally dropped [[phab:T282227|T282227]] * 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]] * 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]] * 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/ * 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip` * 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs === 2021-05-05 === * 16:25 Majavah: add self to sudo policy `roots` * 16:07 arturo: grant `taavi` projectadmin (Majavah) === 2021-05-04 === * 10:47 arturo: rebase & resolve merge conflicts in labs/private.git === 2021-05-03 === * 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]]) === 2021-04-29 === * 18:10 bstorm: added and removed an etcd node === 2021-04-23 === * 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was "notready" for some reason === 2021-04-20 === * 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244 === 2021-04-13 === * 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]]) * 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful) * 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]]) * 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]]) === 2021-04-08 === * 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]] === 2021-04-06 === * 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number ([[phab:T267082|T267082]]) === 2021-04-01 === * 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook) * 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs === 2021-03-31 === * 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]] === 2021-03-24 === * 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM * 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]]) * 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01 * 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]]) === 2021-03-23 === * 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]]) * 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]]) === 2021-03-18 === * 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix * 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project) * 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful * 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]]) * 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456 * 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow * 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]]) * 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here * 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]] * 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]] === 2021-03-17 === * 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]]) * 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]]) * 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]]) * 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]]) * 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]]) === 2021-03-16 === * 12:32 arturo: added packages jobutils / misctools v1.41 to <nowiki>{</nowiki>stretch,buster<nowiki>}</nowiki>-toolsbeta aptly repository in tools-sge-services-03 === 2021-03-11 === * 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]] === 2021-03-10 === * 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor === 2021-02-26 === * 20:39 andrewbogott: rebooting all hosts * 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]]) * 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]]) * 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]]) * 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]]) === 2021-02-19 === * 12:42 arturo: deploying new version of the ingress admission controller * 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta * 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) * 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) === 2021-02-10 === * 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) * 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4 * 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150 (toolsbeta-proxy-3), it was previously pointing to an old IP address === 2021-02-08 === * 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]] === 2021-02-05 === * 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing === 2021-01-21 === * 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]] === 2021-01-13 === * 14:10 dcaro: dcaro doing puppet tests, puppet runs might break * 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]]) * 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]]) === 2020-12-22 === * 10:48 arturo: rebase & resolve ugly git merge conflict in labs/private.git === 2020-12-18 === * 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]]) === 2020-12-14 === * 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]] * 19:25 bstorm: created temporary instance toolsbeta-io-test-local [[phab:T267966|T267966]] === 2020-12-11 === * 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts === 2020-12-10 === * 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]]) === 2020-12-09 === * 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]]) === 2020-12-04 === * 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]]) * 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]]) === 2020-11-23 === * 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]]) * 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]]) * 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]]) * 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]]) * 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]]) * 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]]) * 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]]) === 2020-11-18 === * 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]]) * 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]]) === 2020-11-17 === * 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]]) * 12:09 Lucas_WMDE: <dcaro> 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]]) * 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]]) * 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]]) * 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]]) === 2020-11-16 === * 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]]) * 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]]) === 2020-11-10 === * 19:42 bstorm: safelisted "argocd" namespace with namespaceSelector for registry-admission controller * 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it * 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]]) * 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]]) * 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]]) * 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]]) === 2020-11-06 === * 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]] * 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]] * 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]]) * 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave === 2020-11-05 === * 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]]) * 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]]) === 2020-11-04 === * 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]]) * 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03 * 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use * 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use * 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix === 2020-11-03 === * 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]]) * 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing * 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04 * 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02 === 2020-11-02 === * 13:35 arturo: added dcaro as projectadmin & user ([[phab:T266068|T266068]]) === 2020-10-29 === * 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]]) === 2020-10-28 === * 18:58 andrewbogott: deleting toolsbeta-puppetmaster-03 — seems broken and unused === 2020-10-22 === * 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]] === 2020-09-10 === * 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive) * 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]]) * 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]]) * 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]]) === 2020-09-09 === * 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic. * 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down * 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least * 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them * 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]]) * 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]]) * 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]]) === 2020-07-15 === * 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]] === 2020-07-14 === * 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing === 2020-07-08 === * 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]]) === 2020-06-26 === * 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]]) === 2020-06-24 === * 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]]) * 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]]) * 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]]) * 11:26 arturo: add TXT record `"v=spf1 mx -all"` [[phab:T120225|T120225]] * 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]] === 2020-06-23 === * 13:10 arturo: added herron to the test tool for email testing * 11:36 arturo: removing `benapetr` and adding myself to the test tool * 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix * 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group * 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org * 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert * 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251 === 2020-06-16 === * 22:54 bd808: Building webservice 0.72 === 2020-06-15 === * 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]] * 17:52 bd808: Building webservice 0.71 === 2020-06-12 === * 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]] === 2020-06-11 === * 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there * 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working * 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O<nowiki>{</nowiki>project:toolsbeta<nowiki>}</nowiki>' 'run-puppet-agent'` we are mostly back in business * 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04 * 12:14 arturo: poweroff toolsbeta-puppetmaster-03 * 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 * 11:53 arturo: create VM toolsbeta-puppetmaster-04 * 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults * 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems * 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03 * 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 * 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` === 2020-06-04 === * 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap === 2020-05-28 === * 11:27 arturo: cleanup livehackings * 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]]) * 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 === 2020-05-27 === * 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]]) * 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]]) * 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]]) === 2020-05-26 === * 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]] * 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]]) * 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]]) * 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]]) * 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]]) * 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]]) * 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]] * 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again === 2020-05-21 === * 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py === 2020-05-19 === * 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]]) * 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb` === 2020-05-15 === * 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]] * 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts === 2020-05-14 === * 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]]) * 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]]) === 2020-05-12 === * 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]] * 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]] * 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]] === 2020-05-08 === * 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]]) * 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]]) === 2020-05-07 === * 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) * 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03 * 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) === 2020-05-06 === * 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta * 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]]) === 2020-05-05 === * 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]]) * 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]]) === 2020-05-04 === * 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project === 2020-04-29 === * 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]] === 2020-04-20 === * 14:47 arturo: added joakino to toolsbeta.admin LDAP group * 12:06 arturo: installing tools-webservice v0.68 for testing * 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there * 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project === 2020-04-10 === * 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]] * 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]] * 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up === 2020-04-08 === * 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]] === 2020-03-31 === * 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta === 2020-03-30 === * 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point) * 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use === 2020-03-24 === * 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]] === 2020-03-19 === * 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]] * 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]]) === 2020-03-16 === * 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]] * 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]] * 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]] * 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]] * 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well * 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests * 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]]) === 2020-03-11 === * 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta === 2020-03-09 === * 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]]) * 13:08 arturo: instance quota was full, bump it from 35 to 40 === 2020-03-06 === * 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names === 2020-03-05 === * 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test === 2020-02-27 === * 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing * 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3] * 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers === 2020-02-26 === * 20:35 bstorm_: hard rebooting the grid master for toolsbeta * 20:20 jeh: restart toolsbeta-sgegrid-shadow === 2020-02-18 === * 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists === 2020-02-10 === * 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-02-07 === * 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]] * 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-01-23 === * 03:14 bd808: Demoted projectadmins not listed in the "roots" sudoer policy to project members just to avoid random confusion * 03:06 bd808: Added legoktm to "roots" sudoer policy * 02:53 bd808: Added legoktm as project admin === 2020-01-22 === * 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-<nowiki>{</nowiki>1,2<nowiki>}</nowiki>, source of cronspam === 2020-01-21 === * 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1 * 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]] === 2020-01-17 === * 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]]) * 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me) === 2020-01-16 === * 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out * 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]]) === 2020-01-14 === * 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch === 2020-01-13 === * 16:41 bstorm_: There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps. === 2020-01-10 === * 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing === 2020-01-07 === * 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]] === 2020-01-06 === * 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things === 2020-01-03 === * 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged * 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]]) === 2020-01-02 === * 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly. * 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups === 2019-12-17 === * 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes === 2019-12-16 === * 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file === 2019-12-04 === * 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use * 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use === 2019-12-02 === * 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster * 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]]) === 2019-11-25 === * 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]]) === 2019-11-21 === * 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet) === 2019-11-15 === * 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]] === 2019-11-14 === * 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug === 2019-11-08 === * 19:36 bstorm_: rebooted the proxy server just in case that fixes something. * 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]]) * 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet === 2019-11-07 === * 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]] === 2019-11-06 === * 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]] * 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]] * 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]] * 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]] * 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized) * 17:35 bstorm_: changing some hiera to work with new proxy host * 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]]) === 2019-11-05 === * 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]] === 2019-10-25 === * 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]] * 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2 === 2019-10-23 === * 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]] * 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]]) * 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]]) * 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]]) === 2019-10-22 === * 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]] * 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]] * 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]] * 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}} * 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]] * 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}} * 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]] === 2019-10-21 === * 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}} * 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]] === 2019-10-18 === * 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud` * 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused) * {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}} * {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}} * 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused) === 2019-10-14 === * 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required * 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]] === 2019-10-09 === * 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]]) === 2019-10-08 === * 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]] * 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]] === 2019-10-07 === * 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue === 2019-09-25 === * 23:31 bd808: Updated user list for "roots" sudoer policy * 23:30 bd808: Granted Krenair projectadmin === 2019-09-05 === * {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}} === 2019-08-12 === * 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]] === 2019-08-09 === * 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003 === 2019-07-24 === * 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]] * 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster === 2019-07-18 === * 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-* * 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]]) * 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code * {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}} === 2019-07-17 === * 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]] * 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]] === 2019-07-15 === * 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix * 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]] === 2019-07-03 === * 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]]) * 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]]) * 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]] === 2019-07-02 === * 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]]) * 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap === 2019-07-01 === * 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]] * 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]] * 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]] * 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]] * 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix === 2019-06-28 === * 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm === 2019-06-25 === * 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]] * 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]] === 2019-06-21 === * 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]] === 2019-06-19 === * 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]]) === 2019-06-14 === * 16:24 bstorm_: Manually failed "back" to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta * 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything * 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on. * 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now === 2019-06-06 === * 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]` * 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix * 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix === 2019-06-05 === * 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets) * 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]] * 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]] * 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment) * 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused === 2019-05-24 === * 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon * 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs * 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01 * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01: * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc === 2019-05-07 === * 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix * 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix * 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix === 2019-04-25 === * 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver: toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master === 2019-04-12 === * 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1 * 00:05 andrewbogott: migrating remaining VMs to eqiad1-r === 2019-03-25 === * 18:00 bd808: All Trusty instances shutdown and now in process of deleting * 17:42 bd808: Preparing to shutdown beta Trusty job grid === 2019-03-22 === * 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]] === 2019-03-15 === * 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]]) === 2019-02-20 === * 14:58 andrewbogott: moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003 === 2019-02-14 === * 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002 === 2018-12-04 === * 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/ === 2018-11-26 === * 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03 * 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM === 2018-11-08 === * 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]]) === 2018-10-22 === * 21:20 bstorm_: launched a stretch/sonofgridengine master server === 2018-09-19 === * 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]] * 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]]) * 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb === 2018-07-22 === * 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]] === 2018-07-18 === * 10:46 harej: Deleted toolsbeta-flynn-01 === 2018-07-12 === * 23:06 bstorm_: Got the grid master running === 2018-06-28 === * 16:34 chicocvenancio: adding harej as root for flynn testing === 2018-06-27 === * 22:35 chicocvenancio: add harej as project admin to test Flynn stuff === 2018-06-22 === * 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning * 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions * 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating * 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01 * 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice * 07:56 andrewbogott: someone removed /usr/bin/webservice === 2018-05-15 === * 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]] * 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01 === 2018-05-08 === * 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]] === 2018-05-07 === * 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs * 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01 * 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2> /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)` * 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd === 2018-05-06 === * 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd === 2018-05-05 === * 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]] * 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml` === 2018-05-02 === * 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera * 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]] * 20:31 zhuyifei1999_: nuke webservice instances and rebuild * 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]] === 2018-04-20 === * 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem === 2018-04-19 === * 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :( * 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead * 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org => toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs * 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]] === 2018-03-30 === * 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]] === 2018-03-14 === * 18:07 chicocvenancio: updated paws-beta k8s cluster and nodes to v1.9.4 for [[phab:T189680|T189680]] === 2018-03-05 === * 19:33 chicocvenancio: added Zhuyifei1999 as project admin === 2018-02-09 === * 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]]) === 2017-08-07 === * 14:09 andrewbogott: deleted etcd-k8s-CTEST and k8s-master-CTEST === 2017-04-26 === * 15:38 madhuvishy: add Madhuvishy as projectadmin === 2016-10-07 === * 19:30 valhallasw`cloud: (puppet certs, to be precise) * 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs === 2016-10-04 === * 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for) === 2016-09-08 === * 17:11 bd808: Added BryanDavis (self) to project as admin === 2016-08-29 === * 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck * 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck * 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains === 2016-07-03 === * 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010 === 2016-05-27 === * 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs === 2016-05-26 === * 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host === 2015-10-13 === * 19:21 valhallasw`cloud: started building toolsbeta-bastion. === 2015-09-07 === * 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest... * 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages) === July 4 === * 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22 === July 2 === * 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting === July 1 === * 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it? * 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login === June 7 === * 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo * 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config === May 11 === * 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages === May 2 === * 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code * 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera * 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in * 10:25 valhallasw`cloud: set Hiera variable "elasticsearch::cluster_name": toolsbeta-logstash-eqiad * 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]]) === April 26 === * 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now * 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test === March 31 === * 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources. It can be restarted when needed. === September 20 === * 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006 === July 22 === * 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-) === August 19 === * 12:44 petan: rebooting apache it seems to be frozen === August 4 === * 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-) === July 6 === * 19:42 petan: rebooting login === June 26 === * 08:03 wm-bot: petrb: updating logsplitter === June 24 === * 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info * 13:43 scfc_de: Made scfc root. * 13:42 scfc_de: Created toolsbeta-puppetmaster. * 11:09 YuviPanda: Granted yuvipanda root on toolsbeta === June 21 === * 13:46 wm-bot: petrb: rebooting all servers === June 17 === * 08:31 petan: switching all instances to nfs === June 16 === * 15:37 petan: importing sudo policies of tools * 15:36 petan: importing security groups of tools * 15:36 petan: blah {{SAL|Project Name=toolsbeta}} <noinclude>[[Category:SAL]]</noinclude> 3aqybm5zxqd70kc8yaax3g6qqz4nhsh 2452777 2452776 2026-08-30T20:17:15Z Stashbot 7414 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api 2452777 wikitext text/x-wiki === 2026-08-30 === * 20:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 20:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-08-27 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:54 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 10:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 10:22 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 10:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2026-08-26 === * 09:49 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:14 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:05 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:00 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 08:59 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-08-25 === * 08:54 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logs-api * 08:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 08:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-08-20 === * 15:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:06 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-19 === * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-18 === * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:33 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:11 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-17 === * 15:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:52 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 11:39 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:38 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:38 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2026-08-13 === * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 09:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 08:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 02:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 02:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.32.13 to 1.33.13 (TT433131) * 01:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:57 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T433131|T433131]]) * 01:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13 * 01:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-13 * 01:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-nfs-13 * 01:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-nfs-13 * 01:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 01:47 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 01:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) === 2026-08-12 === * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 21:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T433131|T433131]]) * 20:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 20:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T433131|T433131]]) * 19:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-13 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:41 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 19:05 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 19:05 andrew@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 18:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 18:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.32.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 17:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.33.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:30 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 16:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 16:02 andrewbogott: upgrading to kubernetes 1.33.13 * 16:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.32.13 to 1.33.13 (TT433131) * 15:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.33.13 ([[phab:T433131|T433131]]) * 15:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-bastion-6 ([[phab:T401804|T401804]]) * 12:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 12:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 12:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:23 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:20 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2026-08-06 === * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 14:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2026-08-05 === * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager ([[phab:T408786|T408786]]) * 14:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager ([[phab:T408786|T408786]]) === 2026-08-04 === * 15:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:53 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:51 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:50 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 09:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 09:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-08-03 === * 14:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-harbor-1 * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-harbor-1 * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:25 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 08:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 08:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-31 === * 12:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:53 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:44 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 11:27 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 11:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-07-30 === * 15:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 14:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 10:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2026-07-27 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:14 godog: set 'dumps_use_nfs_lb: true' for prefix-puppet toolsbeta-test-k8s-worker-nfs - [[phab:T432325|T432325]] * 10:30 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all NFS workers ([[phab:T432325|T432325]]) * 10:10 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T432325|T432325]]) * 09:32 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T432325|T432325]]) * 09:29 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T432325|T432325]]) === 2026-07-23 === * 12:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 12:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 11:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-22 === * 01:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-07-21 === * 15:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-20 === * 09:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-07-17 === * 14:29 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 14:20 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 14:19 tlepage@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:19 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-16 === * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-15 === * 09:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-14 === * 16:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:49 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:07 godog: fail back to classic dumps nfs for toolsbeta-test-k8s-worker-nfs prefix * 12:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-harbor' * 12:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-harbor' * 11:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-6 * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-5 * 11:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-5 * 11:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-redis-4 * 11:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-redis-4 * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-cumin-1 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-cumin-1 * 10:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-cumin' * 10:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-cumin' * 08:49 godog: set 'dumps_use_nfs_lb: true' for mail, bastion, static prefixes * 08:22 godog: test reboot all workers * 07:58 godog: test dumps-nfs switchover on toolsbeta-test-k8s-worker-nfs-8 * 07:44 godog: set 'dumps_use_nfs_lb: true' for toolsbeta-test-k8s-worker-nfs prefix * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-13 === * 21:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 21:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:25 tlepage@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 tlepage@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:34 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-8.toolsbeta.eqiad1.wikimedia.cloud * 12:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-9.toolsbeta.eqiad1.wikimedia.cloud * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud * 11:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-redis-7.toolsbeta.eqiad1.wikimedia.cloud === 2026-07-09 === * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 11:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 09:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-08 === * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-07 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-06 === * 17:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-07-02 === * 16:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'tlepage' in role 'member' ([[phab:T427914|T427914]]) * 09:35 godog: carry out nfs tests on toolsbeta-test-k8s-worker-nfs-8 - [[phab:T411248|T411248]] * 08:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-30 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-29 === * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-25 === * 16:37 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:29 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 16:28 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 16:28 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-06-24 === * 16:51 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T429907|T429907]]) * 16:39 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T429907|T429907]]) * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-06-23 === * 08:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-22 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-18 === * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:30 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-ap * 12:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-ap === 2026-06-15 === * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-06-11 === * 13:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T428449|T428449]]) * 12:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T428449|T428449]]) === 2026-06-08 === * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T427918|T427918]]) * 10:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-3 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-2 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-gateway-1 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 10:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 ([[phab:T427918|T427918]]) * 10:03 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 ([[phab:T427918|T427918]]) * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.31.14 to 1.32.13 ([[phab:T427918|T427918]]) * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:14 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:13 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) * 09:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.32.13 ([[phab:T427918|T427918]]) === 2026-06-04 === * 13:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-06-02 === * 15:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 01:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-06-01 === * 19:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-27 === * 07:54 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 07:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-05-21 === * 09:48 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:33 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:29 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:13 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 09:09 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2026-05-20 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logs-api * 23:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 08:27 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 08:26 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 08:26 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 00:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-05-19 === * 23:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 23:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:15 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 11:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-05-18 === * 09:11 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 09:06 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-14 === * 15:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:04 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway ([[phab:T425479|T425479]]) * 10:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway ([[phab:T425479|T425479]]) === 2026-05-13 === * 17:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 16:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:19 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 08:18 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-05-12 === * 23:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 22:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 22:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 10:45 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T423417|T423417]]) * 10:40 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:36 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) === 2026-05-11 === * 17:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:23 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 15:20 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 10:35 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api ([[phab:T423417|T423417]]) * 10:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T423417|T423417]]) * 00:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 00:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config === 2026-05-06 === * 08:57 taavi: renew prometheus kubernetes certificate about to expire * 08:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 08:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system === 2026-05-05 === * 04:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 04:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:49 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-emailer * 01:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-04-28 === * 19:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 15:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 10:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-23 === * 09:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node === 2026-04-21 === * 14:10 taavi: delete all ingress objects from toolsbeta [[phab:T392356|T392356]] * 13:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 13:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli === 2026-04-20 === * 15:19 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2026-04-10 === * 13:57 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 08:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 08:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-09 === * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-04-08 === * 17:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 08:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2026-04-07 === * 23:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 13:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 13:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 07:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 07:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 07:15 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-gateway * 07:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-04-02 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:36 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 10:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-04-01 === * 17:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-03-31 === * 18:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 15:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:20 godog: reboot harbordb1 instance * 12:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 12:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-03-26 === * 17:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 17:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-03-24 === * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 12:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-23 === * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway === 2026-03-19 === * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 14:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-03-17 === * 19:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2026-03-10 === * 18:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 11:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 11:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-09 === * 17:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 15:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 15:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice-cli * 15:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice-cli * 15:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component istio-gateway * 15:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:56 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:55 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:53 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 12:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:42 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 65536 ram * 12:32 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase by 65536 ram * 12:29 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a gateway role in the toolsbeta cluster * 12:22 taavi@cloudcumin1001: Added a new k8s gateway toolsbeta-test-k8s-gateway-1.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster * 12:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-gateway-1 * 12:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-gateway-1 * 11:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a gateway role in the toolsbeta cluster * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a gateway role in the toolsbeta cluster === 2026-03-06 === * 11:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:32 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 11:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2026-03-05 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-gateway * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-gateway * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system * 15:02 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component istio-system * 14:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 11:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:11 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-04 === * 21:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:30 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:30 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:21 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:20 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:19 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:18 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:02 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:02 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:00 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:35 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:34 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:21 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:20 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 13:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 13:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 04:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 04:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 04:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 04:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 04:15 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 04:14 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:04 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:04 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:32 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:31 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:09 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:05 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:47 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:43 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:57 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:57 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 01:50 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:19 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:18 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 01:13 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 01:12 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:53 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:52 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:46 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:45 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:45 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:40 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:39 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:38 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:37 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 00:27 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 00:25 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:09 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 00:04 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) === 2026-03-03 === * 23:55 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:54 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:53 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:51 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:49 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:49 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:37 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:36 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:10 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 23:01 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:55 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:47 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:41 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:40 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:40 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:29 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:08 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:06 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:06 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:03 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:58 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:58 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:56 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 21:44 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:27 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:17 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:15 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 21:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 21:11 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 20:51 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:50 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 20:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 20:01 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 19:41 root@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 18:19 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-admission * 18:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-03-02 === * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 16:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-26 === * 15:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2026-02-25 === * 14:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component gateway-api * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component gateway-api === 2026-02-24 === * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-02-16 === * 09:10 dcaro: restarted the admin tool, was giving error connecting to lighttpd socket === 2026-02-05 === * 18:46 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 18:28 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2026-02-04 === * 15:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 15:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-02-03 === * 08:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2026-01-28 === * 15:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-23 === * 01:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2026-01-22 === * 18:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 17:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2026-01-15 === * 08:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2026-01-12 === * 17:59 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission * 17:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 17:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 17:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 17:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 17:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2026-01-06 === * 14:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 13:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-12, toolsbeta-test-k8s-ingress-9 * 13:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-11, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 13:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-10 ([[phab:T413796|T413796]]) * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=0) for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-11 ([[phab:T413796|T413796]]) * 13:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 13:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.30.14 to 1.31.14 ([[phab:T413796|T413796]]) * 12:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 12:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade to 1.31.14 ([[phab:T413796|T413796]]) * 02:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 02:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:16 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 02:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:15 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 01:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:13 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=97) * 01:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node ([[phab:T375217|T375217]]) * 01:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:57 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 00:56 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=97) * 00:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2026-01-05 === * 23:32 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:20 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 23:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 23:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:40 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 22:38 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) === 2025-12-18 === * 11:13 godog: bump max objects quota to 100k * 11:06 godog: bump object quota to 200G === 2025-12-15 === * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:21 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics === 2025-12-11 === * 16:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-12-04 === * 18:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 18:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:02 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 17:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 17:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:26 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:20 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 15:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T361237|T361237]]) * 15:41 andrewbogott: deleting toolsbeta-test-k8s-etcd-27 and replacing with a Bullseye node for cluster consistency [[phab:T361237|T361237]] === 2025-12-03 === * 19:24 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 18:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T375217|T375217]]) * 17:23 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 17:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=0) * 17:10 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node (exit_code=99) * 14:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.depool_and_remove_node ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T375217|T375217]]) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T375217|T375217]]) === 2025-12-02 === * 22:10 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:10 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:07 root@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:07 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 22:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster * 21:50 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 21:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster === 2025-12-01 === * 16:22 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:04 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-25 === * 13:27 volans: temporarily disabling puppet on the nfs workers to merge gerrit/1210582 * 10:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-prometheus-2 (cluster eqiad1) * 10:21 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-prometheus-2 (cluster eqiad1) * 02:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 02:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 02:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 01:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 01:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-11-24 === * 10:03 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-20 === * 14:13 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 14:06 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 11:27 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 11:27 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 09:47 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 09:47 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing === 2025-11-19 === * 16:50 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component infra-tracing * 16:33 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:23 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:18 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:11 volans@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component infra-tracing * 16:11 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component infra-tracing * 16:10 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 15:54 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 15:54 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:46 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-11-18 === * 17:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:59 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:52 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-11-17 === * 18:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 18:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 18:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-13 === * 15:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 15:14 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-ingress-12 (cluster eqiad1) * 12:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-bastion-7 (cluster eqiad1) * 12:30 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-bastion-7 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:24 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-8 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) * 12:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.instance.stop_start vm toolsbeta-test-k8s-haproxy-7 (cluster eqiad1) === 2025-11-12 === * 13:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-11-10 === * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 21:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 21:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 21:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 21:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 20:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 20:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 19:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 19:46 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-06 === * 15:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-05 === * 19:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-11-04 === * 16:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 03:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 01:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 01:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 01:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 01:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 00:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 00:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 00:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 00:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 00:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 00:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-11-03 === * 22:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 22:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 22:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 22:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 22:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 22:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 22:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 22:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 21:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 21:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 21:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 21:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 20:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 20:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api * 20:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 18:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 11:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 11:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 10:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 10:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-11-01 === * 00:06 bd808: Re-enabled Puppet on toolsbeta-test-k8s-haproxy-7 ([[phab:T408570|T408570]]) === 2025-10-31 === * 19:53 bd808: Disabled Puppet on toolsbeta-test-k8s-haproxy-7 for manual testing of haproxy changes ([[phab:T408570|T408570]]) === 2025-10-30 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 13:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 11:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logs-api * 11:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logs-api === 2025-10-29 === * 18:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T408669|T408669]]) * 13:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T408669|T408669]]) * 12:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico * 11:44 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2025-10-28 === * 19:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:24 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-10-27 === * 18:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:18 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:57 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:10 dcaro: removing taskruns/pipelineruns v1beta1 version from the stored list in the crds ([[phab:T408127|T408127]]) === 2025-10-23 === * 17:33 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:30 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 17:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 17:08 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-22 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 14:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-10-21 === * 11:48 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 11:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 00:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-16 === * 11:20 volans@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:12 volans@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-10-14 === * 14:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-13 === * 14:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-10-08 === * 11:40 godog: nfs back up and has been for some time [[phab:T406688|T406688]] * 10:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:34 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 10:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:19 godog: shut down nfs while investigating [[phab:T406688|T406688]] * 09:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 08:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-10-07 === * 18:56 root@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:56 root@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component registry-admission * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 12:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 11:56 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 11:52 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:46 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 11:44 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 11:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:56 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 08:52 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 08:39 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:39 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:37 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:36 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:35 godog: move nfs from toolsbeta-nfs-4 to toolsbeta-nfs-5 - [[phab:T404584|T404584]] * 08:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 08:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 08:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-10-03 === * 12:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-6 * 12:42 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-6 * 09:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-5 * 09:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-5 === 2025-10-02 === * 12:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) === 2025-10-01 === * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-09-29 === * 10:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) no stuck workers found * 10:35 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers no stuck workers found === 2025-09-26 === * 08:17 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 08:07 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-25 === * 17:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 15:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:39 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 godog: test reboot of toolsbeta-nfs-4 * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-24 === * 17:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:31 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:19 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-11 * 09:17 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 09:11 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 09:10 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9,toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:04 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-nfs-7 * 09:03 filippo@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-14 * 09:03 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14 * 08:42 filippo@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for toolsbeta-bastion-7 * 08:41 filippo@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-bastion-7 * 08:19 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:19 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:09 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 08:05 wmbot~godog@r5: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 08:03 wmbot~godog@r5: START - Cookbook wmcs.nfs.migrate_service * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:02 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-kubeusers * 07:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 06:34 wmbot~godog@r5: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 06:26 wmbot~godog@r5: START - Cookbook wmcs.nfs.add_server === 2025-09-23 === * 15:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-09-22 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-09-18 === * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 10:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-09-16 === * 16:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:56 taavi: delete toolsbeta-sgebastion puppet prefix * 15:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:43 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:42 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-ingress-10 ([[phab:T404721|T404721]]) * 15:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 15:35 wmbot~dcaro@acme: Added a new k8s ingress toolsbeta-test-k8s-ingress-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 15:29 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:24 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 14:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:24 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:24 wmbot~dcaro@acme: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:15 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:12 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:11 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 14:11 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 14:09 wmbot~dcaro@acme: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-worker-nfs-11 ([[phab:T404721|T404721]]) * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 13:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:41 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T404721|T404721]]) * 13:40 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-5 * 13:38 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.worker.depool_and_remove_node for host toolsbeta-test-k8s-worker-nfs-5 * 13:31 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:31 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-test-k8s-worker-nfs-5 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 13:21 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-test-k8s-worker-nfs-5 * 13:20 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot for tools-test-k8s-worker-nfs-5 * 13:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-15 === * 14:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 13:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 12:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 12:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 11:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 11:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-09-10 === * 14:43 godog: reset puppet to production, testing finished * 14:39 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186938 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1186937 * 08:40 volans: add volans to toolsbeta.admin tool * 08:15 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'volans' in role 'member' * 08:15 fnegri@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'volans' in role 'member' === 2025-09-09 === * 13:22 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.vps.instance.force_reboot (exit_code=0) vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 13:22 wmbot~dcaro@acme: START - Cookbook wmcs.vps.instance.force_reboot vm toolsbeta-cumin-1 (cluster eqiad1, project toolsbeta) * 08:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component mainain-kubeusers * 08:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component mainain-kubeusers === 2025-09-08 === * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 15:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api === 2025-09-05 === * 13:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-04 === * 18:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:04 dcaro: bumped the object quota to 100G, like tools === 2025-09-03 === * 17:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 14:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 12:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 11:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 09:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 08:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 08:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-09-02 === * 23:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 16:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 16:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 15:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-09-01 === * 21:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 21:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 21:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 20:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 20:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 19:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 19:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 18:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 18:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 17:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 16:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 15:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:26 dcaro: upgrade finished * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=0) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions (exit_code=99) for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_bastions for toolsbeta-bastion-6.toolsbeta.eqiad1.wikimedia.cloud, toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud ([[phab:T402377|T402377]]) * 14:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 14:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:49 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=0) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:47 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=97) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 ([[phab:T402377|T402377]]) * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses (exit_code=99) for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_ingresses for toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-ingress-9 * 13:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 13:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 13:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:59 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-ingress-9, toolsbeta-test-k8s-ingress-10, toolsbeta-test-k8s-ingress-11, toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:57 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9, toolsbeta-test-k8s-worker-12, toolsbeta-test-k8s-worker-13 * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:47 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=99) for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers for toolsbeta-test-k8s-worker-nfs-10, toolsbeta-test-k8s-worker-nfs-5, toolsbeta-test-k8s-worker-nfs-7, toolsbeta-test-k8s-worker-nfs-8, toolsbeta-test-k8s-worker-nfs-9 * 12:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers (exit_code=0) no stuck workers found * 12:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade_workers no stuck workers found * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 12:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 12:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 12:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.30.14 to 1.30.14 * 11:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 * 11:35 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 11:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 11:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 10:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 09:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 09:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 09:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 09:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 08:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.29.15 to 1.30.14 ([[phab:T402377|T402377]]) * 08:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.run_tests * 08:26 wmbot~dcaro@acme: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=97) * 08:26 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.run_tests * 08:06 dcaro: starting upgrade to v1.30 ([[phab:T402377|T402377]]) === 2025-08-28 === * 08:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-27 === * 18:13 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 18:13 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase * 17:44 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:42 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:39 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 17:38 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-bastion-7.toolsbeta.eqiad1.wikimedia.cloud * 10:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-08-21 === * 10:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:21 dcaro: test for sal === 2025-08-19 === * 16:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:39 dcaro: deploy wmcs-k8s-metrics upgrade ([[phab:T362869|T362869]]) * 09:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-18 === * 20:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 18:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 18:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-14 === * 15:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 15:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 14:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 02:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-13 === * 16:23 dcaro: reverting jobs-api release ([[phab:T401846|T401846]]) === 2025-08-12 === * 16:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config * 15:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component image-config * 09:53 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) , no stuck workers found * 09:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers , no stuck workers found * 09:52 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 09:50 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers (exit_code=0) for * 09:50 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.k8s.reboot_stuck_workers for * 03:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-cli * 03:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli * 03:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 03:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-08-11 === * 12:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 12:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:55 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:14 dcaro: deploying kyverno ([[phab:T394787|T394787]]) === 2025-08-08 === * 06:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 05:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-07 === * 10:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-08-06 === * 17:34 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli ([[phab:T401014|T401014]]) * 17:33 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli ([[phab:T401014|T401014]]) * 17:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:21 bd808@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 17:21 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 17:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 15:25 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api === 2025-08-05 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 16:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 13:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component webservice-cli * 13:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:17 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tool-webservice * 12:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tool-webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component webservice-cli * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component webservice-cli * 12:16 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 08:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:41 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 02:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 02:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 02:27 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 02:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 02:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 02:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 02:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 02:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 02:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 02:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 01:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 01:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 01:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 01:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 01:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 01:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 01:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 01:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 01:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 01:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 00:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 00:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 00:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-08-04 === * 13:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:05 filippo@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) * 13:05 filippo@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' ([[phab:T401091|T401091]]) === 2025-08-01 === * 03:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-cli * 03:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-cli === 2025-07-31 === * 15:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:13 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 04:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 04:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 03:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 03:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 03:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli === 2025-07-30 === * 09:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 08:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-29 === * 16:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:14 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-28 === * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-23 === * 18:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2025-07-14 === * 14:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) ([[phab:T398715|T398715]]) * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase ([[phab:T398715|T398715]]) === 2025-07-10 === * 10:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:31 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 09:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-07-09 === * 17:03 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 14:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:20 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component misctools-cli * 10:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 10:07 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:46 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component misctools-cli * 08:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli * 08:22 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component misctools-cli * 08:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component misctools-cli === 2025-07-08 === * 17:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-07-07 === * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 11:22 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 08:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component logging * 08:19 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-07-03 === * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 13:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-02 === * 10:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 10:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maiantain-kubeusers * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maiantain-kubeusers * 09:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-07-01 === * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:56 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 15:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 15:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 14:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 14:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 13:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 12:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 10:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 09:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-06-26 === * 16:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 12:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-25 === * 17:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:49 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:46 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 09:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-24 === * 16:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:04 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 10:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component logging * 10:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:58 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging * 09:57 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging * 09:57 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component logging === 2025-06-23 === * 15:31 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 15:28 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-19 === * 18:46 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:43 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 17:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-18 === * 14:22 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-06-17 === * 14:33 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:58 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 09:52 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-06-16 === * 17:35 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 17:32 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 17:31 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 17:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:46 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:29 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:00 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:48 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-12 === * 12:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-11 === * 13:32 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:26 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:25 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:15 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:12 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-10 === * 16:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:53 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 16:53 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:12 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 16:01 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:01 wmbot~dcaro@acme: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 15:58 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 15:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:22 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:10 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:04 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:56 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:54 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:38 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:36 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:32 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:28 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:21 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api ([[phab:T394277|T394277]]) * 12:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api ([[phab:T394277|T394277]]) === 2025-06-09 === * 16:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:09 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:13 chuckonwumelu@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:56 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-07 === * 16:49 dcaro: extend the volume toolforge-prometheus-a to 20G === 2025-06-06 === * 18:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 18:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-cli * 18:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-06-05 === * 14:43 chuckonwumelu@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:30 chuckonwumelu@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-06-04 === * 00:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:35 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-06-02 === * 23:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 23:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:06 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:05 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 18:01 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-22 === * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 19:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-6 * 08:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-6 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-5 * 08:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-prometheus-1 * 07:59 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-prometheus-1 === 2025-05-21 === * 13:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-8.toolsbeta.eqiad1.wikimedia.cloud * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-7.toolsbeta.eqiad1.wikimedia.cloud * 13:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 13:12 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-20 === * 18:24 bd808: Made addshore an admin === 2025-05-19 === * 08:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2025-05-16 === * 18:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-prometheus-2.toolsbeta.eqiad1.wikimedia.cloud * 11:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) * 11:35 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.quota_increase === 2025-05-15 === * 08:13 taavi: renew expiring Puppet CA cert === 2025-05-14 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:44 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:49 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2025-05-13 === * 15:30 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 15:19 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-12 === * 19:05 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 18:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 15:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:10 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 11:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 11:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:44 taavi: fix security groups for frontproxy-nginx metricsinfra job * 10:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 08:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 08:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-05-09 === * 22:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 22:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 22:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 22:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component api-gateway * 17:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-05-08 === * 17:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 17:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 17:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 17:10 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 16:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:46 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 15:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:45 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 13:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:54 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 10:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:53 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:51 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:39 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2025-05-07 === * 17:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:44 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 15:37 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 12:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:17 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:06 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 09:36 taavi: remove 'roots' ldap sudo policy [[phab:T392797|T392797]] * 09:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:19 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 09:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-05-06 === * 12:28 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 12:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 11:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-24 === * 18:24 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 18:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli === 2025-04-23 === * 15:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 15:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 15:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 15:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-cli * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 13:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:11 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-21 === * 10:13 taavi: update cluster-info config map to use k8s.svc.toolsbeta.eqiad1.wikimedia.cloud service name [[phab:T262562|T262562]] === 2025-04-17 === * 16:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 16:25 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 16:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-04-16 === * 19:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 19:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 14:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission === 2025-04-15 === * 13:07 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:33 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:28 arturo: added `toolsbeta-tofu` bot account with `member` permissions [[phab:T391474|T391474]] === 2025-04-11 === * 21:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 19:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 19:42 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-09 === * 10:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-04-08 === * 01:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-04-07 === * 20:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 20:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 20:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 20:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 19:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 19:00 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 18:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:49 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:43 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 08:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 07:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 07:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 07:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 06:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 04:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 04:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-04-04 === * 09:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 09:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 09:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:16 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 09:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 09:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 08:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 08:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 07:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 07:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 07:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 07:14 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 07:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 07:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 06:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-31 === * 14:50 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:49 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:45 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:44 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 14:43 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:31 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:30 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:24 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:20 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:14 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:13 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:12 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:11 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:10 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:08 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:07 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:06 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:05 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:04 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 13:02 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 12:08 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:53 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:42 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:13 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-10 from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:10 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:09 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:04 fnegri@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) * 11:03 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.28.14 to 1.29.15 ([[phab:T390212|T390212]]) === 2025-03-25 === * 15:14 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:03 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:29 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:16 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:57 wmbot~dcaro@acme: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:44 wmbot~dcaro@acme: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 08:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2025-03-24 === * 18:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-20 === * 14:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'chuckonwumelu' in role 'member' * 14:04 aborrero@cloudcumin1001: START - Cookbook wmcs.vps.add_user_to_project for user 'chuckonwumelu' in role 'member' === 2025-03-13 === * 22:32 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 22:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 17:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 17:33 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 17:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:21 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:13 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 16:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:49 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:44 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-cli * 16:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 16:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:12 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:26 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-12 === * 19:00 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362868|T362868]]) * 15:56 fnegri@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-builder * 15:55 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 03:17 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api * 03:08 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-11 === * 18:03 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:54 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:46 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api ([[phab:T362868|T362868]]) * 17:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 17:36 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api ([[phab:T362868|T362868]]) * 17:36 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission ([[phab:T362868|T362868]]) * 17:35 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission ([[phab:T362868|T362868]]) * 17:34 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362868|T362868]]) * 17:33 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362868|T362868]]) * 17:32 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362868|T362868]]) * 17:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:01 fnegri@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362868|T362868]]) * 14:51 fnegri@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362868|T362868]]) * 14:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli * 14:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 14:03 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 13:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:45 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component volume-admission * 10:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission === 2025-03-10 === * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 18:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 18:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 18:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 17:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 17:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-03-06 === * 10:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 10:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 09:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-05 === * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-03-04 === * 21:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 21:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 20:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 14:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 11:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 11:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 09:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission === 2025-03-03 === * 17:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:49 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:40 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 12:46 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 11:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 10:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 10:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 09:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2025-02-27 === * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 14:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 14:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2025-02-26 === * 19:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 19:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:35 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 10:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli * 10:16 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-02-24 === * 20:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 20:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 20:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 20:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 20:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-02-19 === * 17:30 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer ([[phab:T320284|T320284]]) * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer ([[phab:T320284|T320284]]) === 2025-02-17 === * 17:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2025-02-06 === * 17:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 15:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 14:19 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 14:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builer * 13:55 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builer * 13:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 13:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 13:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 13:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:55 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 12:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 12:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 12:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 12:35 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2025-02-01 === * 15:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 15:15 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:14 andrewbogott: hard rebooting all VMs for [[phab:T385264|T385264]] * 15:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes === 2025-01-29 === * 01:01 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 00:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2025-01-23 === * 21:04 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 20:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T370245|T370245]]) * 20:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all NFS workers ([[phab:T370245|T370245]]) * 14:21 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 14:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-22 === * 18:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 18:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-21 === * 16:29 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:28 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:21 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 16:18 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 16:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 16:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 16:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:11 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-14 * 15:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-14 * 15:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 15:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T370245|T370245]]) * 12:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-9 * 12:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-8 * 12:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-7 * 12:48 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-5 * 12:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-5 * 12:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for toolsbeta-test-k8s-worker-nfs-10 * 12:40 andrewbogott: rebooting toolsbeta-nfs-3 and then restarting all k8s-nfs workers * 12:38 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for toolsbeta-test-k8s-worker-nfs-10 === 2025-01-20 === * 13:36 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-cli * 13:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-cli === 2025-01-17 === * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-15 === * 04:02 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:39 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:36 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 03:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2025-01-07 === * 00:31 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component calico * 00:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 00:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 00:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 00:15 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 00:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 00:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-metrics * 00:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-metrics * 00:08 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 00:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2025-01-06 === * 23:55 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 23:46 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component maintain-harbor * 23:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 23:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 23:36 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 23:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 23:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 23:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 23:12 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 23:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 16:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 16:22 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 16:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor === 2024-12-13 === * 13:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 13:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 13:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-12-06 === * 07:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 07:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 07:40 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-05 === * 16:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:07 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:00 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:09 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 13:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-12-04 === * 19:37 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 19:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 19:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 17:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 17:23 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 17:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 16:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 16:36 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 16:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 15:37 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 15:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:26 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 14:20 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 14:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 14:14 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 14:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 14:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:59 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 13:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 13:39 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:38 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 13:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 13:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:10 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 01:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 01:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 01:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 00:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-12-03 === * 21:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:46 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:45 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:26 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 21:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:10 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:06 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:05 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 21:02 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component builds-api * 21:01 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:45 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:44 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 20:41 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 20:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:58 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:36 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:20 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 19:11 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:25 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:23 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:10 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:09 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:07 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:06 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:04 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 18:03 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 18:01 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:22 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:20 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 17:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 17:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-29 === * 08:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 08:29 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 08:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:08 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:07 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 07:05 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 07:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:39 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:34 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 06:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 05:00 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:40 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-harbor * 04:13 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor * 03:31 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:28 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:22 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:56 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 00:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:48 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:47 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:34 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:33 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:31 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:27 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 00:15 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-11-25 === * 12:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:03 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 11:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:40 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 10:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 09:20 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-23 === * 07:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder ([[phab:T358225|T358225]]) * 07:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder ([[phab:T358225|T358225]]) === 2024-11-20 === * 15:06 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:56 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 14:54 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 14:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 14:03 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 14:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 12:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 11:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 00:15 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T362867|T362867]]) * 00:09 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T362867|T362867]]) === 2024-11-19 === * 21:44 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 21:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 21:25 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 21:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:43 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 20:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:36 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-emailer * 20:30 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 20:20 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api ([[phab:T362867|T362867]]) * 20:14 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api ([[phab:T362867|T362867]]) * 20:09 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T362867|T362867]]) * 20:04 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T362867|T362867]]) * 20:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:54 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T362867|T362867]]) * 19:47 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:41 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:38 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission ([[phab:T362867|T362867]]) * 19:24 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission ([[phab:T362867|T362867]]) * 19:23 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 19:17 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component ingress-admission ([[phab:T362867|T362867]]) * 19:16 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission ([[phab:T362867|T362867]]) * 15:37 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:10 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-webservice * 10:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-webservice === 2024-11-18 === * 14:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:57 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 10:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer === 2024-11-14 === * 16:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli * 16:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 16:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 16:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 12:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 12:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-12 === * 13:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:41 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 13:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 09:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-11-11 === * 17:34 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 17:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 16:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 16:17 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:04 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 16:04 dcaro@cloudcumin1001: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:01 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 16:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 16:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:27 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component components-api * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:06 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 15:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component wmcs-k8s-metrics * 14:09 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 13:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:43 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 13:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 13:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-11-07 === * 15:56 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:51 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 14:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 14:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 14:30 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 14:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-11-06 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:16 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:15 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 16:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 15:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 07:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 07:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:31 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 07:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 07:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 07:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-11-05 === * 17:12 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 17:06 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics * 08:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 08:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 07:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 07:39 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico === 2024-11-04 === * 16:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 16:15 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 12:38 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 11:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 11:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 11:11 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 11:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 10:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 10:37 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 10:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 10:20 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 09:47 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-30 === * 15:00 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:59 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:58 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:57 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-10 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:54 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-13 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:53 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:52 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-9 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:51 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-8 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:50 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-7 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:49 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:48 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-5 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:33 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:28 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-12 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:20 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 14:14 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) * 13:16 root@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.27.16 to 1.28.14 ([[phab:T362867|T362867]]) === 2024-10-29 === * 09:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project toolsbeta in eqiad1 * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.vps.create_project for project toolsbeta in eqiad1 === 2024-10-16 === * 09:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-15 === * 17:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 16:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 16:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-10-10 === * 08:20 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 08:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-10-09 === * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 09:11 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-10-08 === * 17:43 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:35 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:34 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 17:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 17:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 16:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 16:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 12:59 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld ([[phab:T376710|T376710]]) * 12:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld ([[phab:T376710|T376710]]) * 08:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:08 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain-kubeusers * 08:07 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 08:03 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component maintain_kubeusers * 08:03 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component maintain_kubeusers === 2024-10-04 === * 11:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 11:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 11:37 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 11:31 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-10-03 === * 14:04 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) [[phab:T374908|T374908]] * 14:03 dcaro: deploying tekton upgrade (builds-builder + builds-api https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/531) === 2024-10-01 === * 10:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 10:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 10:06 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 10:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 09:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission === 2024-09-28 === * 00:06 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 00:01 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli === 2024-09-27 === * 23:51 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 23:44 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-26 === * 16:39 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 16:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 15:57 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 15:51 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission ([[phab:T359641|T359641]]) * 15:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission ([[phab:T359641|T359641]]) * 10:20 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli * 10:12 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli * 10:04 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-cli * 09:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component envvars-cli * 07:59 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-cli * 07:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component builds-cli * 07:45 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component toolforge-weld * 07:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:52 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld * 06:44 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component toolforge-weld * 06:43 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component toolforge-weld === 2024-09-25 === * 14:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-10 * 08:26 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 08:24 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 07:46 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 07:15 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-nfs-7 * 07:02 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:55 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:48 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:33 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:32 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:23 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 06:16 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 06:06 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:59 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:50 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:49 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:48 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 05:48 raymond-ndibe@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:38 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:38 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-10 * 05:37 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-10 * 05:33 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:32 raymond-ndibe@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker-nfs role in the toolsbeta cluster * 05:29 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 05:18 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:17 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 05:16 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 05:15 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 04:42 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 04:40 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-24 === * 22:03 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:56 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers ([[phab:T375157|T375157]]) * 21:41 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno ([[phab:T359641|T359641]]) * 21:35 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component kyverno ([[phab:T359641|T359641]]) === 2024-09-21 === * 03:23 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:18 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 * 03:17 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-worker-nfs-7 * 03:12 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-nfs-7 === 2024-09-20 === * 19:30 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico ([[phab:T341066|T341066]]) * 19:26 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 19:24 raymond-ndibe@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico ([[phab:T341066|T341066]]) * 19:19 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico ([[phab:T341066|T341066]]) * 00:30 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) * 00:25 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component wmcs-k8s-metrics ([[phab:T359641|T359641]]) === 2024-09-19 === * 17:41 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:35 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:34 raymond-ndibe@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli ([[phab:T341066|T341066]]) * 17:28 raymond-ndibe@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:27 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli ([[phab:T341066|T341066]]) * 17:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli ([[phab:T341066|T341066]]) * 14:47 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api ([[phab:T341066|T341066]]) * 14:42 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) * 14:10 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api ([[phab:T341066|T341066]]) * 14:05 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api ([[phab:T341066|T341066]]) === 2024-09-11 === * 12:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 12:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 12:26 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 12:26 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 12:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 11:44 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:34 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:34 wmbot~dcaro@urcuchillay: Added a new k8s ingress toolsbeta-test-k8s-ingress-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:15 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 10:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 09:47 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) * 09:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:24 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-13.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 09:09 wmbot~dcaro@urcuchillay: Added a new k8s worker toolsbeta-test-k8s-worker-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 08:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-09-10 === * 14:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:46 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:35 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-6.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) * 14:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 14:21 dcaro@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-5.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster ([[phab:T359641|T359641]]) === 2024-09-09 === * 16:14 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component cert-manager * 16:09 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 14:29 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-11 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-06 === * 09:17 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:14 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:13 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 09:10 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 09:00 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 08:55 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 08:34 wmbot~raymondndibe@wmf3402: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 08:29 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 08:28 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 08:26 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 06:24 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice === 2024-09-05 === * 20:51 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:50 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:37 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component cert-manager * 20:36 wmbot~raymondndibe@wmf3402: START - Cookbook wmcs.toolforge.component.deploy for component cert-manager * 17:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-9 * 17:39 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 17:39 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-12.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 17:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 17:23 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-8 * 17:22 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-8 * 17:18 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-7 * 17:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-7 * 14:53 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) * 14:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.26.15 to 1.27.16 ([[phab:T359641|T359641]]) === 2024-09-04 === * 14:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:57 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:57 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 13:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 13:55 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 13:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 13:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 12:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 11:20 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers * 11:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers === 2024-09-03 === * 20:01 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 19:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 19:47 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer * 19:40 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer * 19:28 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 19:23 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 19:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-admission * 19:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 19:07 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 19:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 18:50 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 16:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:53 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 16:40 wmbot~dcaro@urcuchillay: Added a new k8s control toolsbeta-test-k8s-control-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 16:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 16:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 16:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 15:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 15:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component kyverno * 15:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:58 dcaro@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component kyverno * 14:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:54 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:44 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) * 14:44 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:32 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component kyverno * 14:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component kyverno * 14:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission * 13:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 13:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 13:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 13:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:53 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:50 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:27 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 10:22 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 10:10 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 10:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 09:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-09-02 === * 09:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 09:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 09:34 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission * 09:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component registry-admission * 09:14 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 08:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 08:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-29 === * 16:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 16:19 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder === 2024-08-28 === * 17:22 andrewbogott: shutting down toolsbeta-harbor-2 to (I hope) quiet alerts. Raymond can start this up again when he's back. * 14:04 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.25.16 to 1.26.15 * 14:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.25.16 to 1.26.15 * 14:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.25.16 to 1.26.15 * 13:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.25.16 to 1.26.15 * 13:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.25.16 to 1.26.15 * 13:51 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.25.16 to 1.26.15 * 13:32 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.25.16 to 1.26.15 * 13:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:49 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 12:43 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.25.16 to 1.26.15 * 06:30 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component ingress-nginx * 06:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx * 06:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-nginx * 06:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component ingress-nginx === 2024-08-27 === * 08:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico * 08:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component calico * 08:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 08:27 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico === 2024-08-26 === * 09:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-21 === * 05:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 05:31 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 05:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-builder * 05:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 05:13 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-builder * 05:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-builder * 04:52 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 04:45 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 04:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission * 04:12 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component volume-admission * 04:03 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission * 03:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission * 03:41 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 03:35 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 03:12 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 03:06 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:59 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:55 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 02:53 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 02:48 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:54 wmbot~raymond@ubuntu: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 01:49 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 01:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 01:42 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.run_tests * 01:39 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 01:38 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.component.deploy for component builds-api === 2024-08-13 === * 09:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 09:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:42 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:40 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api === 2024-08-12 === * 15:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 15:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 12:16 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 12:11 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:05 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 12:03 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 12:00 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 12:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice * 11:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 11:36 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 11:35 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:37 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice * 10:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice * 10:24 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api * 10:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 10:01 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api * 09:59 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component jobs-api * 09:50 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:41 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:14 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:09 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component api-gateway * 09:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway * 09:02 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component api-gateway * 09:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component api-gateway === 2024-08-08 === * 16:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api * 16:45 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api * 16:40 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component envvars-api * 16:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api * 16:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api * 16:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component builds-api * 16:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 15:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 15:28 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components * 15:28 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component components * 15:27 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component compontents * 15:27 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.component.deploy for component compontents === 2024-08-06 === * 13:04 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 13:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 13:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 12:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 12:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api === 2024-08-05 === * 18:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:26 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:22 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 18:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 18:05 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 18:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:58 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:56 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:50 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:49 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api * 17:48 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 17:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api * 17:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.component.deploy for component components-api * 16:52 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:55 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:54 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:53 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:52 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:52 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:51 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:41 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:36 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:14 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:14 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:12 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:11 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:11 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:09 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:04 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:03 wmbot~dcaro@urcuchillay: END (ERROR) - Cookbook wmcs.toolforge.run_tests (exit_code=1) * 15:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 15:01 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 15:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:59 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.run_tests (exit_code=0) * 14:58 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:58 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:57 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:50 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:31 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:28 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:27 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 14:17 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.run_tests (exit_code=99) * 14:17 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.run_tests * 13:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 13:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api * 11:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:07 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:07 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:30 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-08-01 === * 15:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 15:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-31 === * 20:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 20:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:52 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:52 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 12:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 12:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 11:22 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 11:22 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 11:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component components-api * 11:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component components-api === 2024-07-30 === * 17:34 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli === 2024-07-29 === * 18:22 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:21 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 18:02 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:01 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 16:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:10 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 16:09 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 16:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:15 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:13 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 15:03 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:43 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-cli * 12:42 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-cli * 12:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-cli * 12:39 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-cli * 12:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-cli * 11:42 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:41 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 11:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:12 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:07 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:04 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:03 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 11:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 11:00 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:21 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:19 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:18 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 10:02 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:59 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:57 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:56 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-cli * 09:55 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:54 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-cli * 09:53 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-cli === 2024-07-25 === * 15:04 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 15:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:05 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 08:04 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-07-24 === * 08:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 06:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 06:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2024-07-23 === * 14:01 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:01 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 12:53 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 12:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:10 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:10 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:58 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-22 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-18 === * 14:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 08:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-17 === * 14:45 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:45 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:33 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:23 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:23 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:54 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:54 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component jobs-api * 08:54 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 08:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 08:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-07-16 === * 16:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:34 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:34 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:14 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 08:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 08:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-15 === * 14:33 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:33 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 07:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-12 === * 10:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:12 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.24.17 to 1.25.16 * 10:09 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:08 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:07 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-ingress-7 from 1.24.17 to 1.25.16 * 10:00 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.24.17 to 1.25.16 * 09:53 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:51 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.24.17 to 1.25.16 * 09:48 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:47 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.24.17 to 1.25.16 * 09:43 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 * 09:42 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.24.17 to 1.25.16 === 2024-07-11 === * 17:46 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.24.17 to 1.25.16 * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.24.17 to 1.25.16 * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:50 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.24.17 to 1.25.16 * 12:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.24.17 to 1.25.16 * 12:39 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.24.17 to 1.25.16 * 12:31 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-worker-4 from 1.24.17 to 1.25.16 * 12:10 arturo: upgrading k8s cluster to 1.25 (control plane) [[phab:T369168|T369168]] * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 12:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.24.17 to 1.25.16 * 11:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 11:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission === 2024-07-10 === * 17:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 16:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 15:58 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 15:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:48 arturo: manually deleted tool-test8 and tool-test8xx k8s namespaces to have them recreated by maintain-kubeusers * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:01 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 10:01 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-07-09 === * 14:20 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 14:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 14:17 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-07-08 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 13:59 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 13:28 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:28 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 13:06 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 13:05 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 12:26 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 12:26 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 11:29 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:36 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:35 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-07-05 === * 12:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 12:33 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 01:42 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 01:41 wmbot~raymond@ubuntu: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component jobs-api * 01:41 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-07-04 === * 17:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 17:04 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component api-gateway * 17:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 12:57 arturo: updating kubelet flags [[phab:T355881|T355881]] * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:38 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 09:32 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:32 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-07-03 === * 12:19 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:19 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:15 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 10:15 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 09:51 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:50 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2024-07-02 === * 17:00 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 17:00 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 16:48 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:48 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:47 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:47 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:46 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component envvars-api * 14:46 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:54 arturo: cleanup extra redundant cert-signing settings from controller-manager arguments * 10:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 10:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:53 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:22 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 09:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 09:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-07-01 === * 15:39 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:39 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 15:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 14:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:14 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 13:14 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 13:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 13:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-06-28 === * 11:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 11:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 09:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:30 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:30 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:24 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-06-27 === * 16:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-26 * 16:37 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-26 * 16:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-25 * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-25 * 15:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component cert-manager * 14:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component cert-manager * 14:53 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:49 andrew@cloudcumin1001: END (ERROR) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=97) for server toolsbeta-test-k8s-etcd-23 * 14:47 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:47 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:44 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 14:43 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 14:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:36 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=99) for server toolsbeta-test-k8s-etcd-23 * 13:47 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-etcd-23 * 10:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:30 arturo: disabled PodSecurityPolicy admission plugin from kubeadm configmap ([[phab:T368142|T368142]]) * 09:28 arturo: disabled PodSecurityPolicy admission plugin from apiserver static pod manifests ([[phab:T368142|T368142]]) * 08:57 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:57 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-26 === * 10:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:17 arturo: deploying toolforge-webservice 0.103.9 ([[phab:T368463|T368463]]) * 09:15 arturo: setting kyverno policies to Enforce ([[phab:T368141|T368141]]) * 09:15 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-25 === * 12:10 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migrate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migrate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-5' * 11:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.migirate_floating_ip (exit_code=0) for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 11:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.migirate_floating_ip for address 185.15.56.33 to server 'toolsbeta-proxy-6' * 09:42 arturo: deploy toolforge-webservice 0.103.8 ([[phab:T362050|T362050]]) * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-24 === * 15:44 arturo: deploy toolforge-webservice 0.103.7 ([[phab:T362050|T362050]]) * 12:09 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:09 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 10:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-21 === * 03:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs (exit_code=0) for server tbd * 02:59 andrew@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_dbinstance_to_ovs for server tbd === 2024-06-20 === * 14:23 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_project_to_ovs * 12:53 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 12:52 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 09:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-19 === * 09:55 arturo: merging k8s HAproxy change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1047113 * 04:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 04:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 04:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 04:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-17 === * 13:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-ingress-7 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-ingress-7 * 12:05 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-worker-10 * 12:04 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-worker-10 * 11:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-test-k8s-haproxy-5 * 11:52 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-test-k8s-haproxy-5 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-legacy-redirector-2 * 11:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-harbor-1 * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-harbor-1 * 11:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetserver-1 * 11:33 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetserver-1 * 11:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-puppetdb-03 * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-puppetdb-03 * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-6 * 11:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-proxy-5 * 11:28 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-proxy-5 * 11:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-prometheus-1 * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-mail-2 * 11:25 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-mail-2 * 11:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-bastion-6 * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-bastion-6 * 10:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-docker-imagebuilder-2 * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-docker-imagebuilder-2 * 10:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-acme-chief-2 * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-static-2 * 10:48 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-static-2 === 2024-06-14 === * 13:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-sgebastion-05 * 13:10 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-sgebastion-05 * 13:09 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.openstack.migrate_server_to_ovs (exit_code=0) for server toolsbeta-redis-1 * 13:08 taavi@cloudcumin1001: START - Cookbook wmcs.openstack.migrate_server_to_ovs for server toolsbeta-redis-1 * 08:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:21 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:21 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-12 === * 17:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 17:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 17:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 16:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 15:22 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 15:21 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:20 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:31 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:31 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:50 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-11 === * 11:35 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:35 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:40 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:26 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-07 === * 11:28 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component kyverno * 11:28 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component kyverno * 11:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:02 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:50 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:50 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-06 === * 14:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:13 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-06-05 === * 16:04 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:04 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:27 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:26 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:44 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:44 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-06-04 === * 16:12 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:18 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:17 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 12:06 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:25 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:05 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:04 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-06-03 === * 16:21 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:21 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:11 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:10 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:35 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:18 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:17 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:25 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 09:25 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 08:56 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 08:34 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:16 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 08:16 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission === 2024-05-30 === * 12:05 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:05 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-29 === * 14:56 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:56 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 07:45 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 03:00 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 03:00 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-28 === * 12:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:49 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:43 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:43 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-27 === * 16:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 16:02 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:54 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:46 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-25 === * 21:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 21:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 20:37 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 20:36 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-23 === * 13:13 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:13 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-05-15 === * 10:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-05-14 === * 13:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:24 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 13:08 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 13:08 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway === 2024-05-10 === * 13:57 taavi: renew k8s prometheus certificate === 2024-05-07 === * 16:18 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:18 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:29 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) * 15:29 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.openstack.cloudvirt.vm_console * 12:07 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:06 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-05-06 === * 11:29 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:29 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 08:12 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 07:13 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:13 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2024-05-04 === * 15:16 taavi: $ sudo docker exec -it striker-toolsbeta.service poetry run python3 manage.py loaddata software_license.json * 14:20 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 14:20 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2024-05-03 === * 15:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:16 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:16 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-30 === * 10:52 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:52 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-26 === * 08:59 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 08:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-04-25 === * 12:55 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:55 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-24 === * 15:25 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:25 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-18 === * 09:23 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:23 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 08:51 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:51 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-15 === * 20:26 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 20:26 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 18:21 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 18:20 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:51 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:50 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 17:31 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 17:30 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:41 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:38 dcaro@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api * 13:38 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 13:36 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:36 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 11:42 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:42 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 11:02 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:02 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:58 dcaro@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:58 dcaro@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 08:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-12 === * 15:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 15:21 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:15 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:14 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:59 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 14:45 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:39 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:17 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:11 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-admission * 10:10 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-admission * 09:31 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:31 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 09:30 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component volume-admisison * 09:30 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admisison * 09:24 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:23 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 05:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:51 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:47 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 04:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 04:35 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 04:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:37 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 03:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 03:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:21 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 02:05 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:28 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 01:19 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 01:18 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component calico * 01:18 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 01:17 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component calico * 01:17 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway * 01:16 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 01:16 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway * 01:15 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission * 01:15 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission * 01:14 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission * 01:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 01:10 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 01:09 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 01:09 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 00:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node === 2024-04-11 === * 23:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 22:18 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:11 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:10 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:01 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:38 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:27 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:05 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:03 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:02 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:02 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:58 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:34 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 19:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:03 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:23 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 17:12 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:06 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:50 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:34 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 16:31 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:23 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:22 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 08:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 08:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 08:37 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 08:37 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-04-10 === * 19:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:45 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 18:43 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:36 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node * 02:56 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 02:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 02:26 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 02:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:41 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 00:16 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 00:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-04-09 === * 23:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 23:07 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 23:07 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:52 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 22:40 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:29 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 22:17 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 22:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 22:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:24 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:08 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 21:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 21:01 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:52 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 20:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 20:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 20:21 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:52 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:42 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 19:28 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:55 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:39 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:17 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 18:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:33 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 13:33 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:57 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:56 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-08 === * 16:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:51 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 15:44 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 10:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-05 === * 12:13 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:13 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-04-03 === * 16:05 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:04 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 14:30 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:29 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-04-02 === * 19:30 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:22 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 19:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 19:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 19:04 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:58 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 18:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:46 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 18:37 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 18:36 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 18:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 18:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:39 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 17:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:14 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:08 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 17:01 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 17:00 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:54 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:15 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:08 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-localdisk * 15:31 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-localdisk * 15:28 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:22 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:16 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:06 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:06 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:59 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:55 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 14:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 14:33 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:26 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:20 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-registry-02 * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-registry-02 === 2024-04-01 === * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 15:18 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 15:11 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 15:00 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:49 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 14:40 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 14:30 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 14:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 14:13 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node === 2024-03-28 === * 17:51 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 17:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 17:04 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=0) * 16:58 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:56 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_etcd_node (exit_code=99) * 16:50 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_etcd_node * 16:33 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=0) * 16:24 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 16:13 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 16:09 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:54 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:53 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:49 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:49 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:36 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:35 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_etcd_node (exit_code=99) * 15:29 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_etcd_node ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 15:19 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.remove_node_from_hiera ([[phab:T349207|T349207]]) * 14:48 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:48 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:46 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:44 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:43 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera (exit_code=0) ([[phab:T349207|T349207]]) * 14:42 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_hiera ([[phab:T349207|T349207]]) * 14:33 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:32 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:30 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:27 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:25 andrew@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster (exit_code=99) * 14:25 andrew@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.etcd.add_node_to_cluster ([[phab:T360699|T360699]]) * 14:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-3 * 14:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-3 * 13:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-proxy-4 * 13:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'toolsbeta-proxy' * 13:11 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:05 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-proxy-5.toolsbeta.eqiad1.wikimedia.cloud * 13:02 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=99) with prefix 'toolsbeta-proxy' * 12:58 taavi@cloudcumin1001: START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'toolsbeta-proxy' === 2024-03-27 === * 12:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-2 * 12:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-2 === 2024-03-26 === * 14:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) * 14:28 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:22 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:22 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:11 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 14:11 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 14:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) * 14:03 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 14:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 14:01 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:56 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:55 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) * 13:54 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server * 13:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 * 13:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 * 13:34 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:33 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:31 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) * 13:31 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.migrate_service * 13:23 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:22 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 13:20 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) * 13:16 taavi@cloudcumin1001: START - Cookbook wmcs.nfs.add_server === 2024-03-25 === * 18:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-legacy-redirector * 18:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-legacy-redirector === 2024-03-22 === * 11:21 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud * 11:19 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-legacy-redirector-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-21 === * 14:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-4 * 14:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-4 * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_haproxy_node (exit_code=0) for node toolsbeta-test-k8s-haproxy-3 * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_haproxy_node for node toolsbeta-test-k8s-haproxy-3 * 12:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 12:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node === 2024-03-20 === * 15:55 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_haproxy_node (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_haproxy_node * 11:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-19 === * 10:21 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-03-18 === * 12:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-static-1 * 12:33 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-static-1 * 11:42 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 11:40 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-acme-chief-2.toolsbeta.eqiad1.wikimedia.cloud * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud * 10:50 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-static-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-03-16 === * 11:09 taavi: reenable puppet on toolsbeta-test-k8s-control-7/8 === 2024-03-15 === * 10:48 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-14 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-docker-imagebuilder-01 * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-docker-imagebuilder-01 * 13:02 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:01 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-7 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 13:00 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:59 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-6 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-4 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:45 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-3 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:44 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-2 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:43 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:42 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-nfs-1 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-11 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:40 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:39 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-10 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 12:27 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-9 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 11:30 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 11:29 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.restart_static_pods (exit_code=99) for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:40 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.restart_static_pods for toolsbeta-test-k8s-control-8 ([[phab:T359638|T359638]]) * 10:36 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.24.17 ([[phab:T359638|T359638]]) * 10:33 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:31 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 10:12 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-8 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:56 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=97) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 09:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) === 2024-03-13 === * 16:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 16:14 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-control-7 from 1.23.17 to 1.27.17 ([[phab:T359638|T359638]]) * 15:49 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=0) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 15:48 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-12 === * 11:15 aborrero@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.prepare_upgrade (exit_code=99) for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) * 11:15 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.prepare_upgrade for cluster toolsbeta upgrade from 1.23 to 1.24 ([[phab:T359638|T359638]]) === 2024-03-11 === * 16:55 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 16:55 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 16:46 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 16:45 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2024-03-07 === * 14:17 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 14:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 13:41 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-05 === * 16:08 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 16:08 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-03-04 === * 17:55 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 17:55 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-03-01 === * 21:14 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 21:13 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2024-02-28 === * 00:39 bd808@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 00:39 bd808@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-26 === * 13:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud * 13:06 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-docker-imagebuilder-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-02-22 === * 13:59 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 13:58 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 10:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-02-21 === * 17:05 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 17:05 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 15:48 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 15:47 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 14:51 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 14:40 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 14:39 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 14:32 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:27 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:26 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:25 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:24 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:20 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component maintain-kubeusers * 13:16 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-20 === * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-control-6 * 13:48 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-6 * 13:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 13:46 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:38 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-11.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 13:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=2) for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-test-k8s-worker-9 * 13:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the toolsbeta cluster * 13:26 taavi@cloudcumin1001: Added a new k8s worker toolsbeta-test-k8s-worker-10.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 13:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 * 11:56 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.drain (exit_code=0) for node toolsbeta-test-k8s-worker-nfs-1 * 11:55 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.k8s.worker.drain for node toolsbeta-test-k8s-worker-nfs-1 === 2024-02-19 === * 18:46 wmbot~raymond@ubuntu: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 18:44 wmbot~raymond@ubuntu: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-02-15 === * 11:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-5 * 11:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeta-test-k8s-control-5 * 11:08 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-5 * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: Added a new k8s control toolsbeta-test-k8s-control-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:58 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:57 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:57 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:56 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:55 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:53 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:46 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-control-8 * 10:45 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-control-8 * 10:44 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-02-13 === * 14:53 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-control-4 * 14:52 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-5 * 14:51 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-4 * 14:50 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-4 * 10:11 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 10:11 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-8.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:04 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-ingress-3 * 10:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-ingress-3 * 09:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the toolsbeta cluster * 09:59 taavi@cloudcumin1001: Added a new k8s ingress toolsbeta-test-k8s-ingress-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:52 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 09:50 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-4.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 09:40 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 09:40 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-8 * 09:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-7 * 09:38 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-7 === 2024-02-12 === * 10:49 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:48 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 10:33 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:32 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2024-02-09 === * 10:32 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 10:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2024-02-08 === * 15:19 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:18 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 15:15 taavi@cloudcumin1001: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud * 11:30 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:30 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-3.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 11:22 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:06 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-6 * 11:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host toolsbeat-test-k8s-worker-6 * 11:05 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeat-test-k8s-worker-6 * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: Added a new k8s worker-nfs toolsbeta-test-k8s-worker-nfs-2.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:53 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 10:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-worker-10 * 10:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.remove_k8s_node for host toolsbeta-test-k8s-worker-10 === 2024-02-06 === * 11:43 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 11:43 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:58 aborrero@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all nodes ([[phab:T356507|T356507]]) * 10:41 aborrero@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes ([[phab:T356507|T356507]]) === 2024-02-05 === * 09:55 arturo: grant myself member and admin privileges === 2024-01-31 === * 13:50 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:49 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-29 === * 13:06 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud * 13:04 wmbot~taavi@runko: START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-mail-2.toolsbeta.eqiad1.wikimedia.cloud === 2024-01-26 === * 10:59 wmbot~taavi@runko: END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a control role in the toolsbeta cluster * 10:59 wmbot~taavi@runko: Added a new k8s control toolsbeta-test-k8s-control-7.toolsbeta.eqiad1.wikimedia.cloud to the cluster * 10:47 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster * 10:43 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a control role in the toolsbeta cluster * 10:42 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a control role in the toolsbeta cluster === 2024-01-25 === * 12:30 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:30 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:28 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:27 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the toolsbeta cluster * 12:24 wmbot~taavi@runko: START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the toolsbeta cluster * 11:01 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:00 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-24 === * 11:31 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the toolsbeta cluster === 2024-01-23 === * 19:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 19:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component wmcs-k8s-metrics * 19:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 10:07 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 10:06 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-19 === * 15:38 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:38 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 12:08 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-17 === * 14:28 wmbot~dcaro@urcuchillay: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:27 wmbot~dcaro@urcuchillay: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-12 === * 09:22 taavi: upgrade prometheus on toolsbeta-prometheus-1 === 2024-01-11 === * 17:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:10 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-09 === * 17:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 17:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2024-01-08 === * 10:47 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 10:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2024-01-05 === * 14:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:50 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 11:49 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 11:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-12-26 === * 19:15 dhinus: hard reboot toolsbeta-bastion-6 as it's unreachable === 2023-12-20 === * 18:51 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:51 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase * 18:47 wm-bot2: fran@wmf3169 END (FAIL) - Cookbook wmcs.openstack.quota_increase (exit_code=99) * 18:47 wm-bot2: fran@wmf3169 START - Cookbook wmcs.openstack.quota_increase === 2023-12-15 === * 13:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T341067|T341067]]) * 13:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T341067|T341067]]) === 2023-12-13 === * 16:23 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=97) * 16:23 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.scale_grid_exec * 14:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder ([[phab:T352774|T352774]]) * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder ([[phab:T352774|T352774]]) * 13:27 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T338142|T338142]]) * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T338142|T338142]]) * 10:44 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T338142|T338142]]) * 10:43 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T338142|T338142]]) * 09:47 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 09:47 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-12-12 === * 12:13 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 12:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) === 2023-12-11 === * 19:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 19:35 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 15:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 15:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 15:23 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-api ([[phab:T352774|T352774]]) * 15:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:36 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api ([[phab:T352774|T352774]]) * 13:35 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api ([[phab:T352774|T352774]]) * 13:32 dcaro: rebooted the bastion-6, did not seem to have network and was failing to mount nfs * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:25 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 13:23 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 13:23 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission ([[phab:T352774|T352774]]) * 13:22 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission ([[phab:T352774|T352774]]) === 2023-12-07 === * 14:46 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:46 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-05 === * 21:08 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 21:07 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 21:07 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 17:37 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 17:37 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors * 09:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-12-04 === * 09:03 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 09:03 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-12-01 === * 15:49 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 15:49 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-11-23 === * 10:35 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 10:34 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-22 === * 11:26 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 11:26 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 09:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 09:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-11-20 === * 15:00 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 15:00 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-17 === * 15:03 taavi@cloudcumin1001: END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all nodes * 15:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.reboot for all nodes * 14:57 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:57 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 taavi@cloudcumin2001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 14:56 taavi@cloudcumin2001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api === 2023-11-09 === * 15:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 10:39 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 10:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-11-01 === * 09:06 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=99) * 09:06 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-30 === * 14:02 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 14:02 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-27 === * 09:41 dcaro: resizing toolsbeta-prometheus-1 to 4 cores, 8Gram * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:21 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console * 09:18 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) * 09:11 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console === 2023-10-26 === * 09:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics * 09:10 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component wmcs-k8s-metrics * 09:09 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component wmcs-k8s-metrics === 2023-10-25 === * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 11:01 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 11:01 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:28 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster * 10:27 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-test-k8s-ingress-6 * 10:27 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-test-k8s-ingress-6 * 10:26 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the toolsbeta cluster * 10:10 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the toolsbeta cluster === 2023-10-23 === * 15:33 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 15:33 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-20 === * 10:37 blancadesal: harbor up again and upgraded from 2.5 to 2.9 ([[phab:T346241|T346241]]) * 10:11 dcaro: taking harbor down for upgrade ([[phab:T346241|T346241]]) === 2023-10-18 === * 12:18 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 12:17 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder === 2023-10-13 === * 13:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:24 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 09:24 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 09:06 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=97) * 09:06 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-12 === * 11:59 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) * 11:59 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors === 2023-10-10 === * 08:17 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 08:17 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:45 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:45 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-09 === * 07:54 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 07:54 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-05 === * 15:16 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 15:16 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api === 2023-10-04 === * 16:53 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 16:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 16:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 16:17 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 14:56 sstefanova@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:55 sstefanova@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 13:04 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api * 13:03 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api * 07:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 07:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api === 2023-10-03 === * 13:04 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 13:03 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api * 11:42 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api * 09:21 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission * 09:20 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission === 2023-09-27 === * 14:13 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component builds-builder * 14:12 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder * 14:07 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder * 12:29 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config * 12:29 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config === 2023-09-25 === * 07:18 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 07:18 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-20 === * 06:34 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 06:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx === 2023-09-19 === * 15:12 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-nginx * 15:11 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-nginx * 14:48 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 14:47 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-15 === * 12:26 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers * 12:26 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers === 2023-09-14 === * 12:10 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api * 12:09 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api * 12:05 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-emailer * 12:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-emailer * 11:59 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission * 11:58 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission * 11:57 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 11:56 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 10:16 dcaro: deploy bulids-api 0.0.96 * 09:17 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 09:16 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission * 08:54 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission * 08:53 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission === 2023-09-13 === * 16:41 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 16:40 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:38 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 12:37 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay END (ERROR) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=97) for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:27 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusersNone ([[phab:T341084|T341084]]) * 10:06 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone * 10:05 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone === 2023-09-11 === * 16:05 dcaro: deploy builds-builder ([[phab:T341084|T341084]]) * 11:36 dcaro: deploy kubernetes-metrics ([[phab:T341084|T341084]]) === 2023-09-06 === * 08:47 arturo: switch project to new DNS recursor via horizon project hiera ([[phab:T345240|T345240]], [[phab:T342621|T342621]]) === 2023-09-05 === * 13:30 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay END (FAIL) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=99) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:29 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:25 wm-bot2: dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admissionNone ([[phab:T341462|T341462]]) * 13:24 wm-bot2: dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admissionNone ([[phab:T341462|T341462]]) === 2023-08-31 === * 15:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_cluster_status (exit_code=0) * 15:41 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 15:38 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_cluster_status * 12:42 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 12:42 wm-bot2: fran@wmf3169 START - Cookbook wmcs.toolforge.grid.get_job_logs * 12:41 wm-bot2: fran@wmf3169 END (PASS) - Cookbook wmcs.toolforge.grid.get_job_logs (exit_code=0) * 09:36 wm-bot2: deployed kubernetes component api-gateway ({{Gerrit|c0faf0f}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 08:41 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:39 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-7 from 1.22.17 to 1.23.17 * 08:38 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-5 from 1.22.17 to 1.23.17 * 08:36 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:34 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-4 from 1.22.17 to 1.23.17 * 08:31 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:30 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-ingress-3 from 1.22.17 to 1.23.17 * 08:28 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:27 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:25 taavi@cloudcumin1001: END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 * 08:24 taavi@cloudcumin1001: START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node toolsbeta-test-k8s-worker-8 from 1.22.17 to 1.23.17 === 2023-08-30 === * 11:18 wm-bot2: toolsbeta-test-k8s-worker-9: upgraded k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:17 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 11:15 wm-bot2: toolsbeta-test-k8s-worker-9: upgrading k8s from 1.22.17 to 1.23.17 ([[phab:T298005|T298005]]) - cookbook ran by arturo@nostromo * 10:05 dcaro: upgrade toolforge-weld to 1.2.1 ([[phab:T344155|T344155]]) * 08:15 taavi: updating toolsbeta k8s cluster to 1.23 to test new cookbooks, [[phab:T298005|T298005]] [[phab:T343869|T343869]] === 2023-08-29 === * 13:06 wm-bot2: deployed kubernetes component jobs-emailer ({{Gerrit|6f9c8cf}}) - cookbook ran by taavi@runko * 13:03 wm-bot2: deployed kubernetes component jobs-api ({{Gerrit|b29193d}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-28 === * 14:54 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|90055b5}}) ([[phab:T344502|T344502]]) - cookbook ran by dcaro@urcuchillay === 2023-08-22 === * 14:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|27328a4}}) ([[phab:T344668|T344668]]) - cookbook ran by taavi@runko === 2023-08-18 === * 13:40 wm-bot2: deployed kubernetes component envvars-api ({{Gerrit|06c26be}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay * 12:30 wm-bot2: deployed kubernetes component builds-api ({{Gerrit|727e6a7}}) ([[phab:T341462|T341462]]) - cookbook ran by dcaro@urcuchillay === 2023-08-17 === * 12:19 dcaro: deploy builds-api builds-api-0.0.85-20230817105952-{{Gerrit|25c2b55f}} === 2023-08-11 === * 09:06 taavi: fixed /etc/hosts on toolsbeta-nfs-2 because '{{fqdn}}' is not a valid fqdn === 2023-07-26 === * 09:30 wm-bot2: deployed kubernetes component image-config ({{Gerrit|06066ba}}) - cookbook ran by taavi@runko === 2023-07-25 === * 12:59 wm-bot2: deployed kubernetes component image-config ({{Gerrit|0eb287a}}) - cookbook ran by taavi@runko === 2023-07-20 === * 14:34 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image ([[phab:T342338|T342338]], [[phab:T321188|T321188]]) * 10:48 arturo: deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 on toolsbeta === 2023-07-18 === * 10:45 arturo: redeploy jobs-emailer into k8s ([[phab:T341084|T341084]]) === 2023-07-13 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|75db740}}) - cookbook ran by taavi@runko === 2023-07-12 === * 12:46 arturo: deployed builds-admission 0.0.63-20230712120152-{{Gerrit|2ef80a7c}} ([[phab:T341084|T341084]]) === 2023-07-04 === * 13:55 taavi: removed floating IP and public dns records for the harbor server === 2023-07-03 === * 19:08 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config.git ({{Gerrit|561b4d9}}) - cookbook ran by taavi@runko * 08:57 wm-bot2: dcaro doing tests - cookbook ran by dcaro@urcuchillay === 2023-06-26 === * 07:49 dcaro: restarting harbor trove DB (in error status) === 2023-06-21 === * 11:48 dcaro: deploy bulids-api 0.2.0 ([[phab:T337025|T337025]]) * 11:48 dcaro: deploy bulids-api 0.2.0 === 2023-06-16 === * 14:28 dcaro: deployed envvars-api 0.0.1 * 07:41 dcaro: deployed latest builds-api 0.1.0 === 2023-06-15 === * 14:05 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by andrew@bullseye === 2023-06-08 === * 11:54 dcaro: powering off toolsbeta-test-k8s-etcd-22 ([[phab:T334644|T334644]]) === 2023-06-07 === * 12:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ed420b}}) - cookbook ran by taavi@runko === 2023-06-01 === * 10:04 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|7e57832}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 09:16 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:11 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0f4076a}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 09:02 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|f1d94f7}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|6c6a27b}}) ([[phab:T336130|T336130]]) - cookbook ran by dcaro@vulcanus * 07:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|3488cfe}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-26 === * 12:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|ef7f103}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/buildpack-admission-controller ({{Gerrit|d567670}}) ([[phab:T337218|T337218]]) - cookbook ran by dcaro@vulcanus === 2023-05-25 === * 08:40 dcaro: releasing toolforge-weld 1.0.0 ([[phab:T337218|T337218]]) === 2023-05-24 === * 12:26 dcaro: deploy latest buildservice ([[phab:T335865|T335865]]) * 12:26 dcaro: deploy latest buildservice ([[phab:T336050|T336050]]) === 2023-05-23 === * 14:40 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|0c7b25b}}) - cookbook ran by fran@wmf3169 === 2023-05-16 === * 14:45 dcaro: deploy builds-api ([[phab:T336225|T336225]]) * 14:43 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|1a725d0}}) - cookbook ran by dcaro@vulcanus * 11:45 dcaro: release toolforge-weld 0.2.0 and toolforge-webservice 0.98 === 2023-05-15 === * 13:31 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-builds-api ({{Gerrit|0277378}}) - cookbook ran by dcaro@vulcanus * 09:22 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller ({{Gerrit|ad5b2b5}}) - cookbook ran by dcaro@vulcanus === 2023-05-09 === * 17:05 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|e89c581}}) - cookbook ran by taavi@runko * 07:27 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 07:24 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2023-05-05 === * 11:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|87937cd}}) - cookbook ran by taavi@runko === 2023-05-01 === * 23:24 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7199a9e}}) - cookbook ran by raymond@ubuntu === 2023-04-30 === * 14:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-19 - cookbook ran by taavi@runko * 14:42 wm-bot2: removed instance toolsbeta-test-k8s-etcd-18 - cookbook ran by taavi@runko * 14:33 wm-bot2: removed instance toolsbeta-test-k8s-etcd-17 - cookbook ran by taavi@runko === 2023-04-19 === * 16:17 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 14:29 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 14:09 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:45 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 13:34 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:52 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:32 wm-bot2: removed instance toolsbeta-test-k8s-etcd-20 - cookbook ran by taavi@runko * 12:10 wm-bot2: removed instance toolsbeta-test-k8s-etcd-21 - cookbook ran by taavi@runko * 12:07 wm-bot2: removed instance toolsbeta-test-k8s-etcd-22 - cookbook ran by taavi@runko === 2023-04-11 === * 14:13 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/volume-admission-controller.git ({{Gerrit|d878e49}}) - cookbook ran by dcaro@vulcanus * 13:29 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|b65439b}}) - cookbook ran by arturo@nostromo * 10:27 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|8f0bfcd}}) - cookbook ran by taavi@runko * 08:59 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-9.toolsbeta.eqiad1.wikimedia.cloud to the cluster - cookbook ran by taavi@runko * 08:46 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko * 08:44 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/calico ({{Gerrit|c6a3e29}}) - cookbook ran by taavi@runko === 2023-04-05 === * 15:53 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 15:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|5ea5992}}) - cookbook ran by taavi@runko * 15:12 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|2be9962}}) - cookbook ran by taavi@runko === 2023-04-03 === * 11:14 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 11:13 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 11:12 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 11:11 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-3 - cookbook ran by arturo@nostromo * 11:10 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-4 - cookbook ran by arturo@nostromo * 11:08 wm-bot2: rebooted k8s node toolsbeta-test-k8s-ingress-5 - cookbook ran by arturo@nostromo * 11:07 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-6 - cookbook ran by arturo@nostromo * 11:05 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 11:03 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-8 - cookbook ran by arturo@nostromo * 11:01 wm-bot2: rebooting the whole toolsbeta k8s cluster (9 nodes) - cookbook ran by arturo@nostromo * 11:00 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:59 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:26 wm-bot2: rebooted k8s node toolsbeta-test-k8s-worker-7 - cookbook ran by arturo@nostromo * 10:24 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 10:22 wm-bot2: rebooted k8s node toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo === 2023-03-19 === * 09:32 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by taavi@runko === 2023-03-14 === * 10:39 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b70adc1}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local * 10:23 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|7d4afeb}}) - cookbook ran by sstefanova@Slavinas-MBP-W.local === 2023-03-13 === * 09:27 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-03-10 === * 16:35 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|8b42b15}}) - cookbook ran by taavi@runko === 2023-03-09 === * 10:08 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|53e7f81}}) - cookbook ran by taavi@runko === 2023-03-07 === * 11:09 taavi: upgrading kubernetes to 1.22 [[phab:T286856|T286856]] === 2023-03-06 === * 12:48 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|6688477}}) - cookbook ran by taavi@runko * 12:45 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|21fef22}}) - cookbook ran by taavi@runko * 12:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/registry-admission-webhook ({{Gerrit|98ce17f}}) - cookbook ran by taavi@runko * 12:00 arturo: delete calico deployment, and try loading it again for https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/1 === 2023-03-05 === * 15:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/labs/tools/maintain-kubeusers ({{Gerrit|3e04025}}) - cookbook ran by taavi@runko === 2023-03-02 === * 11:31 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/toolforge-tool-roles.yaml (https://gerrit.wikimedia.org/r/c/operations/puppet/+/889836) === 2023-03-01 === * 13:15 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13eda9d}}) - cookbook ran by taavi@runko === 2023-02-28 === * 17:18 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|9252af7}}) - cookbook ran by taavi@runko * 17:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e46da83}}) - cookbook ran by taavi@runko * 14:11 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|f90bd8f}}) - cookbook ran by dcaro@vulcanus === 2023-02-23 === * 16:37 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|efb60b3}}) - cookbook ran by taavi@runko * 16:30 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway ({{Gerrit|4e8645a}}) - cookbook ran by taavi@runko === 2023-02-17 === * 11:27 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|eeeea4c}}) - cookbook ran by arturo@endurance * 11:17 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|7729b18}}) ([[phab:T254636|T254636]]) - cookbook ran by arturo@endurance === 2023-02-16 === * 16:01 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:55 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo * 15:28 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/cert-manager ({{Gerrit|d71994e}}) - cookbook ran by arturo@nostromo * 13:47 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/ingress-admission-controller ({{Gerrit|7191997}}) - cookbook ran by taavi@runko * 10:32 arturo: aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml === 2023-02-15 === * 09:30 wm-bot2: cleaned up grid queue errors on toolsbeta-sgegrid-master - cookbook ran by arturo@nostromo === 2023-02-14 === * 20:52 taavi: deploy cert-manager to toolsbeta [[phab:T329453|T329453]] * 12:02 arturo: included tools-manifests 0.25 in toolsbeta-buster aptly repo ([[phab:T329611|T329611]], [[phab:T329467|T329467]], [[phab:T244809|T244809]]) === 2023-02-13 === * 15:03 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|13d87c4}}) - cookbook ran by taavi@runko * 13:55 wm-bot2: drained, depooled and removed worker toolsbeta-test-k8s-worker-5 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Drained node toolsbeta-test-k8s-worker-4 - cookbook ran by arturo@nostromo * 13:46 wm-bot2: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by arturo@nostromo * 13:45 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:31 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:30 wm-bot2: Depooling and removing worker , will pick the oldest - cookbook ran by arturo@nostromo * 13:15 arturo: cordoned & drained k8s workers 4 to 7 to force workload to relocate to 8 ([[phab:T329378|T329378]]) * 12:35 wm-bot2: Added a new k8s worker toolsbeta-test-k8s-worker-8.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by arturo@nostromo * 12:24 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-10 === * 16:14 wm-bot2: Adding a new k8s worker node - cookbook ran by arturo@nostromo === 2023-02-01 === * 15:41 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|372037f}}) - cookbook ran by taavi@runko === 2023-01-26 === * 14:33 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|307f302}}) - cookbook ran by taavi@runko === 2023-01-23 === * 11:26 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d5ae229}}) ([[phab:T311918|T311918]]) - cookbook ran by taavi@runko === 2023-01-20 === * 15:58 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-6 - cookbook ran by arturo@nostromo * 15:56 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-5 - cookbook ran by arturo@nostromo * 15:54 wm-bot2: renewed kubeadm certs on toolsbeta-test-k8s-control-4 - cookbook ran by arturo@nostromo === 2023-01-19 === * 11:46 arturo: `aborrero@toolsbeta-test-k8s-control-4:~$ sudo -i kubectl delete clusterrolebinding jobs-api-psp` (cleanup unused stuff) === 2023-01-18 === * 15:36 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0ad4c66}}) - cookbook ran by arturo@nostromo === 2023-01-17 === * 13:56 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8cf38a1}}) - cookbook ran by arturo@endurance * 13:46 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0d0a882}}) - cookbook ran by arturo@endurance * 13:45 arturo: add login.toolsbeta.wmflabs.org DNS record as CNAME to toolsbeta-sgebastion-05.toolsbeta.eqiad1.wikimedia.cloud === 2023-01-10 === * 11:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8e0a2f9}}) - cookbook ran by arturo@endurance * 10:42 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0243967}}) - cookbook ran by arturo@endurance === 2022-12-09 === * 08:45 dcaro: manually started puppetdb after killed by oom ([[phab:T324812|T324812]]) === 2022-11-30 === * 10:37 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|bc3529d}}) - cookbook ran by arturo@nostromo === 2022-11-29 === * 12:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|864171a}}) - cookbook ran by taavi@runko * 12:22 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|a8b6e17}}) - cookbook ran by taavi@runko * 09:54 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|9528ed3}}) - cookbook ran by taavi@runko === 2022-11-28 === * 18:39 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|ec5c82b}}) - cookbook ran by taavi@runko * 18:36 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config ({{Gerrit|5394a34}}) - cookbook ran by taavi@runko === 2022-11-15 === * 12:40 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 11:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu === 2022-11-14 === * 20:05 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 19:58 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org/test-raymond - cookbook ran by raymond@ubuntu * 14:14 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:14 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 * 14:12 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by fran@wmf3169 === 2022-11-07 === * 13:32 wm-bot2: deployed kubernetes component https://github.com/toolforge/buildpack-admission-controller ({{Gerrit|b4e912e}}) - cookbook ran by fran@wmf3169 === 2022-11-04 === * 12:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d464be4}}) ([[phab:T304900|T304900]]) - cookbook ran by arturo@nostromo === 2022-11-01 === * 12:42 taavi: remove labstore1006/7 from acme-chief-1 fstab and reboot === 2022-10-24 === * 16:42 wm-bot2: rebooted buster webgen grid workers - cookbook ran by andrew@bullseye * 16:29 wm-bot2: rebooting buster webgen grid workers - cookbook ran by andrew@bullseye * 14:54 wm-bot2: Increased quotas by 30 gigabytes - cookbook ran by dcaro@vulcanus === 2022-10-18 === * 10:24 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|64385e9}}) ([[phab:T320405|T320405]]) - cookbook ran by arturo@nostromo === 2022-10-17 === * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:37 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:36 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:35 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:28 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:27 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:25 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:17 wm-bot2: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the lifecycle image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:16 wm-bot2: Updating the bash image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus * 14:14 wm-bot2: Updating the tekton related images on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-10-14 === * 07:53 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|0cc020e}}) - cookbook ran by taavi@runko === 2022-10-12 === * 10:29 dcaro: deploying new registry-admission controller === 2022-10-10 === * 08:41 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|afa90ed}}) ([[phab:T320284|T320284]]) - cookbook ran by taavi@runko === 2022-09-28 === * 09:48 arturo: manually starting gridengine-master.service on toolsbeta-sgegrid-master ([[phab:T318788|T318788]]) === 2022-09-27 === * 14:23 arturo: briefly livehacking puppetmaster === 2022-08-24 === * 11:55 wm-bot2: deployed kubernetes component https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-nginx ({{Gerrit|7d0e951}}) - cookbook ran by taavi@runko === 2022-08-12 === * 07:24 dcaro_away: started postgresql on puppetdb-02, might have crashed during the ceph issues, now puppet runs on toolsbeta work again === 2022-08-03 === * 15:46 dhinus: recreated jobs-api pods to pick up new ConfigMap * 14:51 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|c47ac41}}) - cookbook ran by fran@MacBook-Pro.station === 2022-08-01 === * 14:01 taavi: unbreak acme-chief after keystone communication issues === 2022-07-19 === * 15:45 taavi: deploying and testing maintain-kubeusers updates === 2022-06-28 === * 15:23 wm-bot2: Adding a new k8s worker node - cookbook ran by taavi@runko === 2022-06-24 === * 07:01 wm-bot2: removing grid node toolsbeta-sgewebgrid-lighttpd-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:59 wm-bot2: removing grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:57 wm-bot2: removing grid node toolsbeta-sgeexec-0902.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko * 06:55 wm-bot2: removing grid node toolsbeta-sgeexec-0901.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by taavi@runko === 2022-06-19 === * 16:28 taavi: restart OOM'd puppetdb on toolsbeta-puppetdb-02 === 2022-06-03 === * 13:17 bd808: publish tools-webservice 0.86 ([[phab:T309821|T309821]]) * 05:25 wm-bot2: rebooted buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting buster weblight grid workers - cookbook ran by taavi@runko * 05:20 wm-bot2: rebooting stretch weblight grid workers - cookbook ran by taavi@runko === 2022-05-30 === * 13:42 taavi: run grid-configurator to remove stale config for some removed nodes === 2022-05-26 === * 15:38 wm-bot2: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|e6fa299}}) - cookbook ran by taavi@runko === 2022-04-20 === * 07:53 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|8f37a04}}) ([[phab:T305592|T305592]]) - cookbook ran by taavi@runko === 2022-04-15 === * 13:26 taavi: shutdown toolsbeta-services-01, not exactly sure what it does and it has no roles applied [[phab:T306100|T306100]] === 2022-04-11 === * 14:47 dcaro: deploying custom version of the regitsry admission hook === 2022-04-08 === * 10:45 arturo: disabled debug mode on the k8s jobs-emailer component === 2022-04-05 === * 07:43 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|d7d3463}}) - cookbook ran by arturo@nostromo * 07:21 arturo: deploying toolforge-jobs-framework-cli v7 === 2022-04-04 === * 16:58 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api ({{Gerrit|cbcfc47}}) - cookbook ran by arturo@nostromo * 09:28 arturo: deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions === 2022-03-25 === * 11:31 dcaro: All alerting VMs rebooted, checking that everything is "working" ([[phab:T304672|T304672]]) * 10:55 dcaro: force restarting all the other nfs-bound VMs one by one ([[phab:T304672|T304672]]) * 10:43 dcaro: restarting the sge-shadow ([[phab:T304672|T304672]]) * 10:32 dcaro: restarting the sge-master ([[phab:T304672|T304672]]) === 2022-03-16 === * 15:23 taavi: deploying https://gerrit.wikimedia.org/r/c/cloud/toolforge/volume-admission-controller/+/737171/ as a [[phab:T292238|T292238]] test to toolsbeta === 2022-03-15 === * 17:55 wm-bot: deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-emailer ({{Gerrit|084ee51}}) - cookbook ran by arturo@nostromo === 2022-03-14 === * 16:14 wm-bot: Updating the distroless/base image on docker-registry.tools.wmflabs.org - cookbook ran by dcaro@vulcanus === 2022-03-11 === * 15:55 dcaro: added provisional toolforg cli package to toolsbeta buster repo ([[phab:T299026|T299026]]) * 15:11 dcaro: added tekton cli package to toolsbeta repos ([[phab:T299026|T299026]]) * 15:02 arturo: deploy jobs-framework-emailer {{Gerrit|9470a5f}} ([[phab:T286135|T286135]]) * 11:59 arturo: deploy jobs-framework-emailer {{Gerrit|d60ffd6}} ([[phab:T286135|T286135]]) === 2022-03-08 === * 08:20 taavi: reboot toolsbeta-cumin-1 for kernel updates === 2022-03-07 === * 15:44 dcaro: Deployed buildpack-admission-controller with the latest code ([[phab:T297090|T297090]]) === 2022-02-17 === * 08:16 taavi: made toolsbeta-puppetmaster-04 its own client to fix `puppet node deactivate` puppetdb access === 2022-02-08 === * 13:04 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/760933 ([[phab:T284767|T284767]]) * 12:19 arturo: created puppet prefix `toolsbeta-sgecron` with proper hiera/roles * 12:16 arturo: created VM toolsbeta-sgecron-02 ([[phab:T284767|T284767]]) === 2022-02-04 === * 18:53 taavi: upgrading to kubernetes 1.21 [[phab:T282942|T282942]] === 2022-01-28 === * 16:28 wm-bot: trying to join node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@nostromo === 2022-01-25 === * 11:45 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2022-01-20 === * 12:35 wm-bot: removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 12:34 wm-bot: removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-19 === * 14:11 arturo: craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta === 2022-01-18 === * 15:56 wm-bot: removing grid node toolsbeta-sgewebgrid-generic-0901 (depool/drain, remove VM and reconfigure grid) - cookbook ran by andrew@buster * 15:30 andrewbogott: switching scratch mount over to the cloud-hosted service with git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/43/754043/1 && git cherry-pick FETCH_HEAD * 09:46 arturo: creating VM toolsbeta-sgebastion-05, deleting toolsbeta-bastion-05 (wrong prefix) === 2022-01-17 === * 18:09 wm-bot: pooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@nostromo * 18:07 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo * 17:54 wm-bot: removing grid node toolsbeta-sgewebgen-10-4 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 13:39 wm-bot: pooled grid node toolsbeta-sgeexec-10-5 - cookbook ran by arturo@nostromo === 2022-01-14 === * 11:56 wm-bot: removing grid node toolsbeta-sgewebgen-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 11:49 wm-bot: removing grid node toolsbeta-sgeexec-10-5 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:57 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:53 wm-bot: removing grid node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.org (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo * 09:44 wm-bot: removing grid node toolsbeta-sgeweblight-10-2 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo === 2022-01-12 === * 12:28 wm-bot: created node toolsbeta-sgeweblight-10-1.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo * 11:27 arturo: created puppet prefix `toolsbeta-sgeweblight`, drop `toolsbeta-sgeweblig` * 11:02 arturo: created puppet prefix 'toolsbeta-sgeweblig' * 11:00 wm-bot: created node toolsbeta-sgeexec-10-6.toolsbeta.eqiad1.wikimedia.cloud and added it to the grid - cookbook ran by arturo@nostromo === 2022-01-11 === * 11:11 wm-bot: created a grid exec node toolsbeta-sgeexec-10-5.toolsbeta.eqiad1.wikimedia.cloud - cookbook ran by arturo@nostromo * 09:20 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@nostromo === 2021-12-23 === * 13:32 wm-bot: trying to join node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 12:11 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-10-4.toolsbeta.eqiad1.wikimedia.cloud to the pool - cookbook ran by arturo@endurance * 11:58 wm-bot: node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster in toolsbeta. - cookbook ran by arturo@endurance * 11:40 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 11:26 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:25 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2 to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:24 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:59 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:34 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 10:31 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance === 2021-12-22 === * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:02 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 12:01 wm-bot: reconfiguring the grid by using grid-configurator - cookbook ran by arturo@endurance * 11:24 wm-bot: removing instance toolsbeta-sgewebgen-09-1 - cookbook ran by arturo@endurance * 11:21 wm-bot: removing grid node toolsbeta-sgewebgen-09-1 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@endurance * 11:19 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance * 10:42 wm-bot: depooled grid node toolsbeta-sgewebgen-10-1 - cookbook ran by arturo@endurance === 2021-12-21 === * 16:32 wm-bot: removing instance toolsbeta-sgewebgen-10-2 - cookbook ran by arturo@endurance * 16:24 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 16:24 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:50 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:07 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:04 wm-bot: Node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:04 wm-bot: Joining grid node toolsbeta-sgewebgen-10-3.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 12:03 wm-bot: Node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud joined the grid cluster toolsbeta. - cookbook ran by arturo@endurance * 12:03 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:48 wm-bot: Joining grid node toolsbeta-sgewebgen-10-2.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance * 11:06 arturo: bump quotas, instances from 50 to 55, CPU from 100 to 150, RAM from 200GB to 250GB ([[phab:T277653|T277653]]) === 2021-12-16 === * 12:46 wm-bot: Joining grid node toolsbeta-sgewebgen-10-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by arturo@endurance === 2021-12-15 === * 14:03 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:31 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance * 13:29 wm-bot: Adding a new grid webgrid generic node - cookbook ran by arturo@endurance === 2021-12-08 === * 05:15 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1028 === 2021-11-28 === * 17:44 andrewbogott: moving toolsbeta-test-k8s-etcd-17 to cloudvirt1019; cloudvirt1018 (its old host) has a degraded raid which is affecting performance === 2021-11-16 === * 12:37 majavah: testing calico 3.21 upgrade [[phab:T292698|T292698]] === 2021-11-05 === * 19:07 majavah: testing registry-admission changes === 2021-10-28 === * 12:48 arturo: update ingress-nginx via helm for `--watch-ingress-without-class=true` === 2021-10-25 === * 14:41 majavah: deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image [[phab:T292771|T292771]] === 2021-10-20 === * 12:15 majavah: upload toolforge-webservice 0.78 to stretch,buster,bullsye-toolsbeta repositories === 2021-10-16 === * 07:47 majavah: deployed cert-manager and wave as a test for automating [[phab:T292238|T292238]] === 2021-10-14 === * 15:02 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:01 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Joining grid node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the toolsbeta cluster - cookbook ran by dcaro@vulcanus === 2021-10-13 === * 11:18 wm-bot: Added a new grid webgrid generic node toolsbeta-sgewebgen-09-1.toolsbeta.eqiad1.wikimedia.cloud to the pool ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:19 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-12 === * 16:10 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:46 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:05 majavah: start gridengine-master.service on toolsbeta-sgegrid-master === 2021-10-11 === * 15:24 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:32 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-07 === * 14:21 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 14:06 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 13:31 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:55 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 08:04 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 07:58 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-06 === * 10:36 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:13 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:08 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:07 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus * 10:05 wm-bot: Adding a new grid webgrid generic node ([[phab:T292465|T292465]]) - cookbook ran by dcaro@vulcanus === 2021-10-04 === * 17:07 bstorm: reboot everything [[phab:T291406|T291406]] * 17:06 bstorm: use cumin to edit fstab to remove old nfs mounts [[phab:T291406|T291406]] * 16:41 bstorm: setting mount_nfs: true on toolsbeta-mail prefix (which is the correct setting) * 14:45 dcaro: rebooting toolsbeta-sgewebgrid-generic-0901.toolsbeta.eqiad1.wikimedia.cloud to force a fsck of the dm-0 device on boot ([[phab:T290970|T290970]]) === 2021-10-01 === * 12:34 arturo: rebooting toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) * 12:12 arturo: experimenting with newer mono runtime on toolsbeta-sgebastion-04 ([[phab:T292289|T292289]]) === 2021-09-29 === * 22:13 bstorm: ran label fix script to use new label format * 22:12 bstorm: toollabs-webservice 0.77 deployed === 2021-09-28 === * 10:32 majavah: removing all podpreset objects and disabling settings.k8s.io/v1alpha1 api === 2021-09-27 === * 16:13 majavah: testing volume-admission fix for containers with some volumes mounted === 2021-09-23 === * 17:14 majavah: testing new maintain-kubeusers release [[phab:T279106|T279106]] === 2021-09-22 === * 18:07 bstorm: launching toolsbeta-nfs-test-client-01 to run a "fair" test battery against [[phab:T291406|T291406]] === 2021-09-15 === * 08:04 majavah: tools-manifest 0.24, [[phab:T290325|T290325]] === 2021-09-14 === * 15:45 majavah: disable podpreset admission plugin in toolsbeta [[phab:T279106|T279106]] * 11:42 arturo: deploying jobs-framework-emailer {{Gerrit|3045601}} ([[phab:T286135|T286135]]) * 10:44 arturo: deploying jobs-framework-emailer {{Gerrit|51032af}} ([[phab:T286135|T286135]]) * 10:39 arturo: deploying jobs-framework-api {{Gerrit|16fbf51}} ([[phab:T286135|T286135]]) === 2021-09-13 === * 15:44 majavah: deploy volume-admission-controller in background; [[phab:T279106|T279106]] === 2021-09-09 === * 17:36 bstorm: deploying a base tekton triggers setup [[phab:T267374|T267374]] * 16:50 majavah: enable unattended updates on toolsbeta [[phab:T290494|T290494]] * 16:19 arturo: {{Gerrit|70017ec0ac}} root@toolsbeta-test-k8s-control-4:~# kubectl apply -f /etc/kubernetes/psp/base-pod-security-policies.yaml * 00:26 bstorm: deleted toolsbeta-sgeexec-0902 since it had a badly screwed up /tmp === 2021-09-03 === * 22:34 bstorm: backfilled quotas for [[phab:T286784|T286784]] === 2021-08-30 === * 23:23 bstorm: deleting toolsbeta-workflow-test [[phab:T289709|T289709]] === 2021-08-21 === * 00:17 bstorm: rebooting the control plane nodes for kubernetes because it can't make things worse [[phab:T289390|T289390]] === 2021-08-20 === * 23:19 bstorm: tried renewing all the certs to get certs working again in kubernetes === 2021-08-12 === * 16:55 bstorm: deployed updated manifest for ingress-admission * 15:02 majavah: deploying ingress-admission-controller using v1 api [[phab:T280436|T280436]] === 2021-07-30 === * 08:01 majavah: replace toolsbeta-sgeexec-1002 with -1004 for [[phab:T287666|T287666]] === 2021-07-29 === * 14:08 majavah: add mdipietro as projectadmin [[phab:T287287|T287287]] * 13:06 majavah: rebuild toolsbeta-sgeexec-1001 as -1003 [[phab:T287666|T287666]] === 2021-07-23 === * 13:31 majavah: upgrading toolsbeta to kubernetes 1.19, [[phab:T280340|T280340]] === 2021-07-22 === * 15:32 arturo: re-deploying toolforge-jobs-framework-api === 2021-07-21 === * 11:58 arturo: deploying jobs-framework-api {{Gerrit|07346d715d17585db9c16dd152cc91ef0bea33c3}} ([[phab:T286108|T286108]]) * 10:51 arturo: enabling TTLAfterFinished feature gate on static pod manifests on /etc/kubernetes/manifests/kube-<nowiki>{</nowiki>apiserver,controller-manager<nowiki>}</nowiki>.yaml in all 3 control nodes ([[phab:T286108|T286108]]) * 10:47 arturo: enabling TTLAfterFinished feature gate on kubeadm live configmap ([[phab:T286108|T286108]]) * 10:09 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/705848 === 2021-07-20 === * 21:18 bstorm: applied `login_server: true` to toolsbeta-sgecron-01 [[phab:T287037|T287037]] * 19:09 bstorm: upgraded version of maintain-kubeusers to the latest in master branch [[phab:T285011|T285011]] * 08:36 majavah: resolve merge conflicts on labs/private === 2021-07-16 === * 19:53 bstorm: set matchPolicy to equivalent on ingress admission controller for toolsbeta [[phab:T280360|T280360]] * 14:04 arturo: deployed jobs-framework-api {{Gerrit|42b7a88}} ([[phab:T286132|T286132]]) === 2021-07-15 === * 15:39 arturo: deploy toolforge-jobs-framework-api git version {{Gerrit|d85d93ee1c5d4be6a526cf83e806b2679dde3875}} === 2021-07-14 === * 09:05 majavah: testing calico 3.18 upgrade - [[phab:T280342|T280342]] === 2021-07-12 === * 11:42 majavah: rebooting toolsbeta-sgeexec-1002, nfs issues === 2021-07-07 === * 09:48 majavah: set dummy values for openstack ldap user/pass hiera values for disable_tool manifests to work === 2021-07-01 === * 17:01 majavah: updating jobs-framework-api * 10:00 arturo: refreshed jobs-api deployment === 2021-06-29 === * 09:28 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:28 wm-bot: Drained node toolsbeta-test-k8s-worker-3. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Draining node toolsbeta-test-k8s-worker-3... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:27 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:13 wm-bot: Drained node toolsbeta-test-k8s-worker-2. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Draining node toolsbeta-test-k8s-worker-2... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:12 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 09:09 wm-bot: Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus * 09:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:59 wm-bot: Drained node toolsbeta-test-k8s-worker-1. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:58 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Draining node toolsbeta-test-k8s-worker-1... ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus * 08:57 wm-bot: Depooling and removing worker , will pick the oldest. ([[phab:T267140|T267140]]) - cookbook ran by dcaro@vulcanus === 2021-06-28 === * 14:46 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooled and removed worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Drained node toolsbeta-test-k8s-worker-4. - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Depooling and removing worker toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud. - cookbook ran by dcaro@vulcanus * 13:23 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:22 wm-bot: Draining node toolsbeta-test-k8s-worker-4... - cookbook ran by dcaro@vulcanus * 13:16 wm-bot: Draining node toolsbeta-test-k8s-worker-4.toolsbeta.eqiad1.wikimedia.cloud... - cookbook ran by dcaro@vulcanus * 11:30 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:25 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:23 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:12 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 11:06 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:54 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:53 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:44 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 10:11 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 09:16 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:51 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 08:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-25 === * 15:27 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:21 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:17 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:15 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:08 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:07 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:03 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:02 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:00 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:45 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:19 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:18 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:57 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:56 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:55 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 13:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:50 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 12:26 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-24 === * 15:52 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:33 dcaro: created flavor g3.cores4.ram8.disk20.ephem40 for the k8s workers * 15:10 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 15:09 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:59 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:35 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:31 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:28 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:24 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus * 14:13 wm-bot: Adding a new k8s worker node - cookbook ran by dcaro@vulcanus === 2021-06-22 === * 18:24 majavah: rolling out kubernetes patch release 1.18.20, cluster is currently at 1.18.18 === 2021-06-17 === * 11:44 majavah: toolsbeta-puppetdb-02: stop puppetdb to free up its ram usage, start postgres process, start puppetdb up again === 2021-06-16 === * 15:53 majavah: add default security group rule allowing prometheus01.metricsinfra to connect to node-exporter port 9100 === 2021-06-15 === * 16:10 majavah: set toolsbeta-bastion-05 as grid submit host === 2021-06-14 === * 21:29 bstorm: deploy package with the staged patch to switch away from os.execv to QA in toolsbeta as toollabs-webservice version 0.75 [[phab:T282975|T282975]] * 10:19 arturo: deploying toolforge jobs-framework-api in kubernetes (just a test) ([[phab:T283238|T283238]]) === 2021-06-12 === * 14:42 majavah: sync hiera key prometheus_nodes to match tools === 2021-06-11 === * 15:25 majavah: undeploy nginx-ingress-jobs from kubernetes * 14:54 majavah: generate and add own root key to passwords::root::extra_keys === 2021-06-08 === * 15:11 majavah: updating k8s worker nodes to 1.18 [[phab:T280299|T280299]] * 15:02 majavah: continuing to update k8s ingress nodes [[phab:T280299|T280299]] * 14:57 majavah: continuing to update rest of k8s control nodes [[phab:T280299|T280299]] * 14:42 majavah: remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work * 14:08 majavah: update toolsbeta-test-k8s-control-4 to kubernetes 1.18 [[phab:T280299|T280299]] === 2021-06-03 === * 16:55 majavah: renew ingress-admission-controller certificates [[phab:T280301|T280301]] * 16:49 majavah: renew registry-admission-webhook certificates [[phab:T280301|T280301]] === 2021-05-25 === * 17:14 andrewbogott: deleting old ingress controllers toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 * 17:13 andrewbogott: created two new ingress nodes, toolsbeta-test-k8s-ingress-4 and toolsbeta-test-k8s-ingress-5 * 15:09 dcaro: turning off VM toolsbeta-test-k8s-etcd-14 to be able to reboot cloudvirt1020 === 2021-05-24 === * 19:40 andrewbogott: replacing existing etcd nodes with localdisk nodes === 2021-05-19 === * 11:35 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/692875/ * 06:51 Majavah: depool toolsbeta-test-k8s-ingress-1 === 2021-05-15 === * 07:52 Majavah: set profile::wmcs::kubeadm::control::apiserver_cert_alternative_names hiera key and adjust config map [[phab:T262562|T262562]] === 2021-05-14 === * 11:22 arturo: allowed VIP address from the new port 172.16.3.26 into the ports of toolsbeta-redis-[1-3] ([[phab:T153810|T153810]]) * 11:16 arturo: aborrero@cloudcontrol1005:~ $ sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-redis-vip ([[phab:T153810|T153810]]) === 2021-05-13 === * 08:07 Majavah: creating toolsbeta-redis-[1-3] as g3.cores1.ram2.disk20 to experiment with redis-sentinel / [[phab:T153810|T153810]] === 2021-05-10 === * 19:42 bstorm: setting profile::wmcs::kubeadm::docker_vol: false on ingress nodes * 17:43 Majavah: testing https://gerrit.wikimedia.org/r/c/operations/puppet/+/688361 in toolsbeta [[phab:T264221|T264221]] * 11:50 Majavah: testing ingress-nginx update https://gerrit.wikimedia.org/r/c/operations/puppet/+/685715 on toolsbeta [[phab:T264221|T264221]] === 2021-05-08 === * 10:42 Majavah: create new ingress node toolsbeta-k8s-ingress-3 [[phab:T264221|T264221]] === 2021-05-07 === * 17:00 bstorm: deleted "toolsbeta-test-k8s-haproxy-2", "toolsbeta-test-k8s-haproxy-1" when the dns caches finally dropped [[phab:T282227|T282227]] * 16:30 bstorm: recreated k8s.toolsbeta.eqiad1.wikimedia.cloud. as a CNAME to k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. [[phab:T282227|T282227]] * 16:16 Majavah: create record k8s.svc.toolsbeta.eqiad1.wikimedia.cloud. pointing to haproxy vip [[phab:T282227|T282227]] * 14:20 Majavah: cherry pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/686607/ * 09:44 arturo: `sudo wmcs-openstack --os-project-id=toolsbeta port create --network lan-flat-cloudinstances2b toolsbeta-k8s-haproxy-keepalived-vip` * 08:19 Majavah: rebuild toolsbeta-test-k8s-haproxy-[12] without nfs === 2021-05-05 === * 16:25 Majavah: add self to sudo policy `roots` * 16:07 arturo: grant `taavi` projectadmin (Majavah) === 2021-05-04 === * 10:47 arturo: rebase & resolve merge conflicts in labs/private.git === 2021-05-03 === * 13:23 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 ([[phab:T278109|T278109]]) === 2021-04-29 === * 18:10 bstorm: added and removed an etcd node === 2021-04-23 === * 17:24 bstorm: rebooting toolsbeta-test-k8s-control-6 because it was "notready" for some reason === 2021-04-20 === * 19:01 bstorm: updated the maintain-kubeusers:beta image to https://gerrit.wikimedia.org/r/c/labs/tools/maintain-kubeusers/+/680244 === 2021-04-13 === * 16:41 arturo: create VM toolsbeta-sgeexec-1002 ([[phab:T277653|T277653]]) * 15:44 arturo: delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful) * 15:36 arturo: created VM toolsbeta-sgeexec-0903 (buster) ([[phab:T277653|T277653]]) * 15:31 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ ([[phab:T277653|T277653]]) === 2021-04-08 === * 18:27 bstorm: cleaned up the deprecated entries in /data/project/.system_sge/gridengine/etc/submithosts for toolsbeta-sgegrid-master and toolsbeta-sgegrid-shadow using the old fqdns [[phab:T277653|T277653]] === 2021-04-06 === * 13:11 dcaro: Removing etcd member toolsbeta-test-k8s-etcd-7.tools.eqiad1.wikimedia.cloud to get an odd number ([[phab:T267082|T267082]]) === 2021-04-01 === * 15:17 dcaro: etcd cluster shrunk 3 members (using wmcs.toolforge.remove_etcd_node cookbook) * 14:54 dcaro: shrinking etcd cluster to 3 members, cleaning up automation runs === 2021-03-31 === * 18:22 bstorm: redeploy ingress-admission controller with `kubectl apply -k deploys/toolsbeta` from the repo [[phab:T275478|T275478]] === 2021-03-24 === * 12:17 arturo: attach the `toolsbeta-docker-registry-data` volume to the `toolsbeta-docker-registry-02` VM * 11:41 arturo: created VM toolsbeta-docker-registry-02 as Debian buster ([[phab:T278303|T278303]]) * 11:34 arturo: attached cinder volume `toolsbeta-docker-registry-data` as /dev/vdb on toolsbeta-docker-registry-01 * 11:23 arturo: created 2G cinder volume `toolsbeta-docker-registry-data` ([[phab:T278303|T278303]]) === 2021-03-23 === * 11:22 arturo: drop and build again the VM toolsbeta-sgregrid-master ([[phab:T277653|T277653]]) * 11:07 arturo: drop and build again the VM toolsbeta-sgregrid-shadow ([[phab:T277653|T277653]]) === 2021-03-18 === * 18:55 bstorm: set profile::toolforge::infrastructure across the entire project with login_server set on the bastion prefix * 18:50 arturo: deleting VMs toolsbeta-paws-worker-1001 toolsbeta-paws-worker-1002 toolsbeta-paws-master-01 (testing for PAWS should happen in the paws project) * 18:49 arturo: deleting VM toolsbeta-workflow-test, no longer useful * 18:44 arturo: replacing toolsbeta-sgegrid-master with a Debian Buster VM ([[phab:T277653|T277653]]) * 16:24 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/672456 * 12:53 arturo: create anti-affinity server group toolsbeta-sgegrid-master-shadow * 12:51 arturo: rebuild toolsbeta-sgegrid-shadow instance as debian buster ([[phab:T277653|T277653]]) * 12:50 arturo: added puppet prefix `toolsbeta-sgegrid-shadow`, migrate puppet config from VM to here * 12:48 arturo: destroy VM toolsbeta-buster-gridmaster (no longer useful) [[phab:T277653|T277653]] * 12:47 arturo: delete puppet prefix `toolsbeta-buster-grirdmaster` (no longer useful) [[phab:T277653|T277653]] === 2021-03-17 === * 12:39 arturo: created VM toolsbeta-buster-gridmaster ([[phab:T277653|T277653]]) * 12:38 arturo: created puppet prefix 'toolsbeta-buster-gridmaster' ([[phab:T277653|T277653]]) * 12:00 arturo: create VM toolsbeta-buster-sgeexec-01 ([[phab:T277653|T277653]]) * 11:56 arturo: created puppet prefix 'toolsbeta-buster-sgeexec' ([[phab:T277653|T277653]]) * 10:34 arturo: re-create toolsbeta-bastion-05 ([[phab:T275865|T275865]]) === 2021-03-16 === * 12:32 arturo: added packages jobutils / misctools v1.41 to <nowiki>{</nowiki>stretch,buster<nowiki>}</nowiki>-toolsbeta aptly repository in tools-sge-services-03 === 2021-03-11 === * 12:33 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/667144 for [[phab:T275865|T275865]] === 2021-03-10 === * 16:48 arturo: briefly stopping VM toolsbeta-test-k8s-etcd-8 to migrate hypervisor === 2021-02-26 === * 20:39 andrewbogott: rebooting all hosts * 15:35 dcaro: removed toolsbeta-test-k8s-etcd-9 with depool from kubeadmin/etcd ([[phab:T274497|T274497]]) * 11:46 arturo: `openstack server create --os-project-id toolsbeta --image debian-10.0-buster --flavor g2.cores2.ram4.disk40 --network lan-flat-cloudinstances2b --property description='buster bastion test' toolsbeta-bastion-05` ([[phab:T275865|T275865]]) * 11:39 arturo: created puppet prefix 'toolsbeta-bastion' to hold new configuration for buster-based bastions ([[phab:T275865|T275865]]) * 09:09 dcaro: Playing around with cookbooks by adding/removing etcd nodes, etcd might missbehave from time to time ([[phab:T274497|T274497]]) === 2021-02-19 === * 12:42 arturo: deploying new version of the ingress admission controller * 11:46 arturo: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) which should only affect toolsbeta * 10:27 arturo: create DNS record `jobs.svc.toolsbeta.eqiad1.wikimedia.cloud` with CNAME to `k8s.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) * 10:25 arturo: create DNS zone `svc.toolsbeta.eqiad1.wikimedia.cloud` ([[phab:T274139|T274139]]) === 2021-02-10 === * 12:34 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/662941 ([[phab:T274139|T274139]]) * 12:23 arturo: add `webserver` security group to toolsbeta-proxy-3 and -4 * 12:20 arturo: fix A record for `toolsbeta.wmflabs.org`, point it to 172.16.1.150 (toolsbeta-proxy-3), it was previously pointing to an old IP address === 2021-02-08 === * 11:48 arturo: trying to introduce TLS support in the front proxy [[phab:T274123|T274123]] === 2021-02-05 === * 00:36 bstorm: updated jobutils and miscutils to 1.40 in aptly for toolsbeta testing === 2021-01-21 === * 15:29 bstorm: pushed the maintain-kubeusers:beta tag with the new code to the docker repo [[phab:T271847|T271847]] === 2021-01-13 === * 14:10 dcaro: dcaro doing puppet tests, puppet runs might break * 10:07 arturo: allocate floating IP 185.15.56.84, and use it for docker-registry.toolsbeta.wmflabs.org (instance toolsbeta-docker-registry-01) ([[phab:T271867|T271867]]) * 10:05 arturo: release and delete floating IP 185.15.56.242 (docker-registry.toolsbeta.wmflabs.org) ([[phab:T271867|T271867]]) === 2020-12-22 === * 10:48 arturo: rebase & resolve ugly git merge conflict in labs/private.git === 2020-12-18 === * 10:52 arturo: live-hacking local puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/650470 ([[phab:T267966|T267966]]) === 2020-12-14 === * 19:27 bstorm: create temporary instance toolsbeta-test-io-unthrottled [[phab:T267966|T267966]] * 19:25 bstorm: created temporary instance toolsbeta-io-test-local [[phab:T267966|T267966]] === 2020-12-11 === * 23:31 bstorm: increasing the output throttle for toolsbeta-test-k8s-haproxy-* nodes in order to figure out what's up with the timeouts === 2020-12-10 === * 08:58 dcaro: starting a new etcd instance completely from ansible playbook (etcd-8) ([[phab:T267412|T267412]]) === 2020-12-09 === * 15:30 dcaro: Playing aronud adding a new etcd node (k8s-etcd-7) ([[phab:T267412|T267412]]) === 2020-12-04 === * 11:17 dcaro: Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity ([[phab:T267412|T267412]]) * 10:12 dcaro: Trying to create a whole new etcd member from ansible ([[phab:T267412|T267412]]) === 2020-11-23 === * 14:17 dcaro: All control nodes re-imaged ([[phab:T267140|T267140]]) * 14:08 dcaro: Taking control-3 node out as control-6 is up and running ([[phab:T267140|T267140]]) * 11:12 dcaro: Launching control-6, to replace control-3 ([[phab:T267140|T267140]]) * 10:45 dcaro: Taking out control-2 node, replaced by control-5 (I saw one 503 reply on the proxy when creating control-5, fyi) ([[phab:T267140|T267140]]) * 10:32 dcaro: Creating new control-5 node (will replace control-2) ([[phab:T267140|T267140]]) * 09:58 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267140|T267140]]) * 09:57 dcaro: Remove control-1 node from the pool (was replaced by control-4) ([[phab:T267195|T267195]]) === 2020-11-18 === * 11:46 dcaro_: Modifying the security groupts to mirror tools ([[phab:T267140|T267140]]) * 10:50 dcaro_: Adding new control-4 node to the control cluster ([[phab:T267140|T267140]]) === 2020-11-17 === * 15:32 dcaro: Creating new toolsbeta-test-k8s-control-4 node and adding it to the cluster ([[phab:T267140|T267140]]) * 12:09 Lucas_WMDE: <dcaro> 11:59:36 UTC – toolbeta up and running again, documented on the live doc for now, apsrever had the wrong config ([[phab:T267140|T267140]]) * 10:40 arturo: hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml in all 3 k8s control nodes to account for new etcd servers ([[phab:T267140|T267140]]) * 08:58 dcaro: etcd hosts reimaged ([[phab:T267140|T267140]]) * 08:54 dcaro: etcd-4,5 and 6 are up and running, removing 1,2 and 3 ([[phab:T267140|T267140]]) === 2020-11-16 === * 11:44 dcaro: etcd5 member added, creating instance toolsbeta-test-k8s-etcd6 and adding to the etcd cluster ([[phab:T267140|T267140]]) * 11:27 dcaro: Creating instance toolsbeta-test-k8s-etcd5 and adding to the etcd cluster ([[phab:T267140|T267140]]) === 2020-11-10 === * 19:42 bstorm: safelisted "argocd" namespace with namespaceSelector for registry-admission controller * 18:49 legoktm: associated floating IP to toolsbeta-docker-registry-01 and pointed DNS docker-registry.toolsbeta.wmflabs.org. at it * 18:27 legoktm: creating toolsbeta-docker-imagebuilder-01 ([[phab:T267616|T267616]]) * 17:18 dcaro: launching instance toolsbeta-test-k8s-etcd-4 ([[phab:T267140|T267140]]) * 17:15 dcaro: removing unused toolsbeta-k8s-etcd prefix (we use toolsbeta-test-k8s-etcd) ([[phab:T267140|T267140]]) * 14:44 dcaro: taking down one of the test-k8s etcd nodes to reimage ([[phab:T267140|T267140]]) === 2020-11-06 === * 23:44 bstorm: toolsbeta k8s cluster fully upgraded to 1.17.13 [[phab:T263284|T263284]] * 21:23 bstorm: upgrading toolsbeta-test-k8s-control-1 to k8s 1.17.13 [[phab:T263284|T263284]] * 15:56 dcaro: Deleting instances proxy-1 and proxy-2, that will finish the proxy rebuild ([[phab:T267140|T267140]]) * 15:53 dcaro: Removing proxy-1 and proxy-3 from hiera, proxy-3 stays as active and proxy-4 as backup ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave ([[phab:T267140|T267140]]) * 13:18 dcaro: bringin up a new proxy-4 instance as slave === 2020-11-05 === * 16:40 dcaro: Moving active proxy from proxy-1 to proxy-3 ([[phab:T267140|T267140]]) * 15:54 dcaro: Adding toolsbeta-proxy-3 to the list of slave proxies in hiera ([[phab:T267140|T267140]]) === 2020-11-04 === * 15:42 dcaro: re-creating the toolsbeta-proxy-03, used wrong image on the first try ([[phab:T267140|T267140]]) * 15:21 dcaro: creating new proxy instance toolsbeta-proxy-03 * 15:18 arturo: dropping project hiera config for `toollabs::checker_hosts`, `toollabs::proxy::ssl_certificate_name`, `toollabs::proxy::ssl_install_certificate` and `toollabs::proxy::web_domain`, no longer in use * 15:16 arturo: dropping project hiera config for `toollabs::proxy::proxies`, no longer in use * 11:46 dcaro: The k8s scheduler-01 fails to connect to etcd (not sure ever did), trying to fix === 2020-11-03 === * 16:04 arturo: add dcaro to the toolsbeta.admin LDAP group ([[phab:T266068|T266068]]) * 15:30 dcaro: [[phab:T267121|T267121]]: Puppetmaster replaced, also removed old puppetdb master from hiera, testing * 15:07 dcaro: Replacing old puppetmaster 02 and 03 from hiera with 04 * 10:55 dcaro: dcaro investigating puppet errors on toolsbeta-puppetdb-02 === 2020-11-02 === * 13:35 arturo: added dcaro as projectadmin & user ([[phab:T266068|T266068]]) === 2020-10-29 === * 22:20 legoktm: switched test tool over to use buildpack image ([[phab:T265681|T265681]]) === 2020-10-28 === * 18:58 andrewbogott: deleting toolsbeta-puppetmaster-03 — seems broken and unused === 2020-10-22 === * 16:22 bstorm: created buildpack psp for [[phab:T265557|T265557]] === 2020-09-10 === * 09:17 arturo: force-rebooting toolsbeta-test-haproxy-2 (unresponsive) * 09:15 arturo: livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/626133 ([[phab:T250172|T250172]]) * 09:00 arturo: tainted/labeld toolsbeta-test-k8s-ingress-1 (and -2) in the k8s cluster ([[phab:T250172|T250172]]) * 08:59 arturo: added toolsbeta-test-k8s-ingress-1 (and -2) to the k8s cluster ([[phab:T250172|T250172]]) === 2020-09-09 === * 11:50 arturo: after force-rebooting everything, the k8s cluster seems to have recovered itself. magic. * 11:45 arturo: force-rebooting the 3 k8s etcd nodes. They seem down * 11:42 arturo: actually, the whole k8s cluster seems down? the API seems down at least * 11:39 arturo: all 3 k8s control nodes seem in bad shape. Wont let me ssh in, or use the console access. Try force-rebooting them * 11:27 arturo: created 2 VMs: toolsbeta-test-k8s-ingress-1 and toolsbeta-test-k8s-ingress-2 ([[phab:T250172|T250172]]) * 11:25 arturo: created new server group toolsbeta-k8s-ingress ([[phab:T250172|T250172]]) * 11:24 arturo: created new puppet prefix `toolsbeta-test-k8s-ingress` ([[phab:T250172|T250172]]) === 2020-07-15 === * 21:35 bstorm: set all of toolsbeta to mount NFS 4.2 except the bastion [[phab:T257945|T257945]] === 2020-07-14 === * 22:28 bstorm: rebooting toolsbeta-sgebastion-04 during NFS testing thing === 2020-07-08 === * 11:08 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/610029 ([[phab:T234617|T234617]]) === 2020-06-26 === * 12:12 arturo: puppetmaster live-hacking with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/608005 ([[phab:T120210|T120210]]) === 2020-06-24 === * 12:55 arturo: live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607279 ([[phab:T120225|T120225]]) * 12:23 arturo: live-hacking puppetmaster with exim prometheus stuff ([[phab:T175964|T175964]]) * 11:31 arturo: live-hack the puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/607320 ([[phab:T175964|T175964]]) * 11:26 arturo: add TXT record `"v=spf1 mx -all"` [[phab:T120225|T120225]] * 11:24 arturo: fix MX record for toolsbeta.wmflabs.org (missing trailing dot) [[phab:T120225|T120225]] === 2020-06-23 === * 13:10 arturo: added herron to the test tool for email testing * 11:36 arturo: removing `benapetr` and adding myself to the test tool * 11:02 arturo: setting `profile::toolforge::mail_domain: toolsbeta.wmflabs.org` in toolsbeta-mail puppet prefix * 10:55 arturo: allow ingress smtp/smtps traffic in the MTA security group * 10:52 arturo: created MX record pointing to mail.toolsbeta.wmflabs.org * 09:43 arturo: restarted nginx in toolsbeta-acme-chief-01 to pickup new certificate, otherwise clients won't accept its TLS cert * 09:38 arturo: live-hacking toolsbeta-puppetmaster-04 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/607251 === 2020-06-16 === * 22:54 bd808: Building webservice 0.72 === 2020-06-15 === * 21:54 bstorm_: removed killgridjobs.sh from toolsbeta bastion [[phab:T157792|T157792]] * 17:52 bd808: Building webservice 0.71 === 2020-06-12 === * 19:41 bstorm_: set `profile::wmcs::nfsclient::mode: soft` on toolsbeta-workflow-test [[phab:T127559|T127559]] === 2020-06-11 === * 12:42 arturo: introduce puppet profile 'toolsbeta-docker-registry' and relocate some hiera config there * 12:39 arturo: for the record, k8s etcd servers certificate changed (puppet based) and k8s just kept working * 12:35 arturo: according to `aborrero@cloud-cumin-01:~$ sudo cumin --force -x 'O<nowiki>{</nowiki>project:toolsbeta<nowiki>}</nowiki>' 'run-puppet-agent'` we are mostly back in business * 12:14 arturo: try switching all VMs to toolsbeta-puppetmaster-04 * 12:14 arturo: poweroff toolsbeta-puppetmaster-03 * 12:12 arturo: copy over labs/private from toolsbeta-puppetmaster-03 to toolsbeta-puppetmaster-04 * 11:53 arturo: create VM toolsbeta-puppetmaster-04 * 11:35 arturo: try reinstalling the python3 stack in toolsbeta-puppetmaster-03, because everything python-related segfaults * 11:33 arturo: reboot toolsbeta-puppetmaster-03 to try cleaning up potential kernel/filesystem problems * 11:32 arturo: apparently every python script segfaults in toolsbeta-puppetmaster-03 * 11:27 arturo: puppetdb wasn't the problem. The problem is puppet-enc segfaulting in toolsbeta-puppetmaster-03 * 11:21 arturo: puppet not working bc puppetdb, run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` === 2020-06-04 === * 21:06 andrewbogott: added krenair to toolsbeta.admin group in ldap === 2020-05-28 === * 11:27 arturo: cleanup livehackings * 10:31 arturo: livehacking puppetmaster and toolsbeta-proxy-1 to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 ([[phab:T253816|T253816]]) * 10:30 arturo: livehacking puppetmaster to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/599139 === 2020-05-27 === * 12:02 arturo: the k8s cluster is now running v1.16.10 ([[phab:T246122|T246122]]) * 11:05 arturo: trying `modules/kubeadm/files/wmcs-k8s-node-upgrade.py --control toolsbeta-test-k8s-control-1 --project toolsbeta --domain eqiad.wmflabs --src-version 1.15 --dst-version 1.16.10 -n toolsbeta-test-k8s-worker-1 -n toolsbeta-test-k8s-worker-2 -n toolsbeta-test-k8s-worker-3` ([[phab:T246122|T246122]]) * 11:02 arturo: upgraded the rest of the k8s control plane nodes to 1.16.10 ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo apt-get install kubelet -y` in the 1.16 version from the component repo ([[phab:T246122|T246122]]) * 10:58 arturo: running `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` and this time it works! ([[phab:T246122|T246122]]) === 2020-05-26 === * 16:17 bstorm_: fix incorrect volume name in kubeadm-config [[phab:T246122|T246122]] * 15:02 arturo: first k8s upgrade failed for yet-to-be-known reasons ([[phab:T246122|T246122]]) * 14:54 arturo: `aborrero@toolsbeta-test-k8s-control-1:~ $ sudo -i kubeadm upgrade apply v1.16.10` ([[phab:T246122|T246122]]) * 14:54 arturo: bump installed version of kubeadm and kubectl to 1.16.10 ([[phab:T246122|T246122]]) * 09:57 arturo: installing kubectl/kubeadm 1.16.9 on k8s worker nodes ([[phab:T246122|T246122]]) * 09:56 arturo: installing kubectl/kubeadm 1.16.9 on k8s control nodes ([[phab:T246122|T246122]]) * 09:30 arturo: set `profile::wmcs::kubeadm::component: 'thirdparty/kubeadm-k8s-1-16'` at project level for trying [[phab:T246122|T246122]] * 09:25 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` broken puppet in this project because puppetdb is down again === 2020-05-21 === * 22:14 bd808: Building tools-webservice 0.70 via wmcs-package-build.py === 2020-05-19 === * 12:20 arturo: trying to install tesseract 4.1.0 in toolsbeta-sgebastion-04 ([[phab:T247422|T247422]]) * 10:18 arturo: `aborrero@toolsbeta-puppetdb-02:~$ sudo systemctl restart puppetdb` === 2020-05-15 === * 20:48 bstorm_: found an error in the new version of maintain-kubeusers, removing the deployment for now [[phab:T246059|T246059]] * 20:35 bstorm_: updating the maintain-kubeusers image to be able to control admin accounts === 2020-05-14 === * 12:09 arturo: created puppet prefix toolsbeta-acme-chief in horizon ([[phab:T252762|T252762]]) * 12:08 arturo: created toolsbeta-acme-chief-01 VM ([[phab:T252762|T252762]]) === 2020-05-12 === * 18:35 bstorm_: upgraded to using typha and rolled back to not doing so -- no affect on existing network [[phab:T250863|T250863]] * 17:44 bstorm_: set the calico version to v3.14.0 because the new liveness probe isn't compatible with the old version. [[phab:T250863|T250863]] * 17:36 bstorm_: deployed an updated bit of yaml for calico without upgrading the version first [[phab:T250863|T250863]] === 2020-05-08 === * 12:48 arturo: allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` ([[phab:T120225|T120225]]) * 12:24 arturo: added puppet prefix `toolsbeta-email` ([[phab:T120225|T120225]]) === 2020-05-07 === * 16:33 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594945 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) * 12:36 arturo: cleanup livehacks in toolsbeta-puppetmaster-03 * 11:12 arturo: livehack toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594925 and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/594926 ([[phab:T251297|T251297]] and [[phab:T250866|T250866]]) === 2020-05-06 === * 19:11 bstorm_: updated toollabs-webservice to 0.69 for toolsbeta * 09:58 arturo: livehacking toolsbeta-puppetmaster-03 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594471 ([[phab:T251297|T251297]]) === 2020-05-05 === * 10:04 arturo: add herron as user and projectadmin, we will work on the email setup ([[phab:T120225|T120225]]) * 09:59 arturo: created VM toolsbeta-mail-01 ([[phab:T120225|T120225]]) === 2020-05-04 === * 13:02 arturo: `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb.service` trying to bring back puppetdb, which is preventing puppet agent runs in the whole project === 2020-04-29 === * 19:48 bstorm_: ran the scary rewrite-psp-preset.sh script across toolsbeta [[phab:T247455|T247455]] === 2020-04-20 === * 14:47 arturo: added joakino to toolsbeta.admin LDAP group * 12:06 arturo: installing tools-webservice v0.68 for testing * 11:05 arturo: poweroff `toolsbeta-services-01`. I suspect this VM is not in use because no puppet role is in used there * 10:58 arturo: run `aborrero@toolsbeta-puppetdb-02:~ $ sudo systemctl restart puppetdb` the service was in failed state, causing puppet failures across the whole project === 2020-04-10 === * 19:32 bstorm_: deployed webservice 0.67 [[phab:T249843|T249843]] * 18:59 bstorm_: delete toolsbeta-gitlab-01 and build toolsbeta-workflow-test [[phab:T249946|T249946]] * 00:40 bd808: REbooting toolsbeta-sgebastion-04. NFS seemed messed up === 2020-04-08 === * 01:10 bstorm_: upgrade toollabs-webservice to 0.66 for qa [[phab:T249390|T249390]] === 2020-03-31 === * 23:39 bstorm_: deployed toollabs-webservice-0.65 to toolsbeta === 2020-03-30 === * 10:35 arturo: remove local changes in the puppet tree in toolsbeta-puppetmaster-03 (docker mount point) * 10:30 arturo: remove puppet prefixes `toolsbeta-test-proxy`, `toolsbeta-k8s-master`, `toolsbeta-flannel-etcd`, no longer in use === 2020-03-24 === * 18:45 jeh: cleanup and remove toolsbeta-elastic7-[1,2,3] VMs (re-configuring hypervisor for local storage) [[phab:T243327|T243327]] === 2020-03-19 === * 23:18 Krenair: Shut down toolsbeta-puppet(db-01{{!}}master-02) - [[phab:T241719|T241719]] * 19:20 arturo: live-hacking toolsbeta-proxy-1 with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/579952 ([[phab:T234617|T234617]]) === 2020-03-16 === * 21:38 bstorm_: removed lots of hiera related to the legacy k8s cluster [[phab:T246689|T246689]] * 19:45 bstorm_: deleting toolsbeta-worker-1001, toolsbeta-k8s-master, toolsbeta-flannel-etcd-01 and toolsbeta-k8s-etcd-01 [[phab:T246689|T246689]] * 19:07 bstorm_: shutting down toolsbeta-flannel-etcd-01 [[phab:T246689|T246689]] * 19:06 bstorm_: shutting down toolsbeta-worker-1001, toolsbeta-k8s-master and toolsbeta-k8s-etcd [[phab:T246689|T246689]] * 14:37 arturo: live-hacking the toollabs-webservice package in toolsbeta-sgewebgrid-lighttpd-0901 as well * 14:22 arturo: live-hacking the toollabs-webservice package in toolsbeta*-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 14:22 arturo: live-hacking the toollabs-webservice package in tools-sgebastion-04 with https://gerrit.wikimedia.org/r/c/operations/software/tools-webservice/+/578413 ([[phab:T234617|T234617]]) * 13:49 arturo: deleting 50 jobs of the `test` tool in the grid to leave room for other tests * 13:18 arturo: live-hack toolsbeta-puppetmaster-02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/578406 ([[phab:T234617|T234617]]) === 2020-03-11 === * 21:32 bstorm_: deployed jobutils_1.39 and miscutils_1.39 to toolsbeta === 2020-03-09 === * 13:11 arturo: created VM `toolsbeta-legacy-redirector` ([[phab:T247236|T247236]]) * 13:08 arturo: instance quota was full, bump it from 35 to 40 === 2020-03-06 === * 16:22 bstorm_: updating maintain-kubeusers image to filter invalid tool names === 2020-03-05 === * 21:22 bstorm_: updated maintain-kubeusers to the latest version for toolsbeta only to live test === 2020-02-27 === * 19:19 bstorm_: upgraded toollabs-webservice to 0.64 on stretch-toolsbeta for testing * 16:03 jeh: create 3 new VMs toolsbeta-elastic7-0[1,2,3] * 16:00 jeh: increase CloudVPS quota instance count for new elasticsearch servers === 2020-02-26 === * 20:35 bstorm_: hard rebooting the grid master for toolsbeta * 20:20 jeh: restart toolsbeta-sgegrid-shadow === 2020-02-18 === * 23:20 bstorm_: added toolsbeta-sgegrid-master.toolsbeta.eqiad1.wikimedia.cloud and toolsbeta-sgegrid-shadow.toolsbeta.eqiad1.wikimedia.cloud to gridengine admin host lists === 2020-02-10 === * 21:19 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.62 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-02-07 === * 23:07 bstorm_: upgraded toollabs-webservice for stetch toolsbeta to 0.60 [[phab:T244611|T244611]] * 21:09 bstorm_: upgraded toollabs-webservice package for stretch toolsbeta to 0.59 [[phab:T244293|T244293]] [[phab:T244289|T244289]] [[phab:T234617|T234617]] [[phab:T156626|T156626]] === 2020-01-23 === * 03:14 bd808: Demoted projectadmins not listed in the "roots" sudoer policy to project members just to avoid random confusion * 03:06 bd808: Added legoktm to "roots" sudoer policy * 02:53 bd808: Added legoktm as project admin === 2020-01-22 === * 11:59 arturo: remove toolviews scripts from toolsbeta-proxy-<nowiki>{</nowiki>1,2<nowiki>}</nowiki>, source of cronspam === 2020-01-21 === * 12:49 arturo: cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1 * 09:40 arturo: livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing [[phab:T234617|T234617]] === 2020-01-17 === * 12:52 arturo: livehack toolsbeta-puppetmaster-02 to test https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/565556 ([[phab:T234617|T234617]]) * 10:37 arturo: enabling puppet agent in toolsbeta-proxy-1 which was disabled without reason since 2019-12-02 (probably by me) === 2020-01-16 === * 23:13 bstorm_: updated toollabs-webservice to 0.58 for stretch to test things out * 12:07 arturo: live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 ([[phab:T242719|T242719]]) === 2020-01-14 === * 02:15 andrewbogott: rebooting toolsbeta-sgecron-01 and toolsbeta-test-k8s-etcd-3 to get nfs unstuch === 2020-01-13 === * 16:41 bstorm_: There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps. === 2020-01-10 === * 21:05 bstorm_: updated toollabs-webservice package to 0.55 for testing === 2020-01-07 === * 15:51 bstorm_: changed kubeadm-config to use a list instead of a hash for extravols on the apiserver in the new k8s cluster [[phab:T242067|T242067]] === 2020-01-06 === * 21:42 bstorm_: disabled rpcbind on toolsbeta-sgebastion-04 to test some things === 2020-01-03 === * 17:46 bstorm_: stashed uncommitted changes on the puppetmaster because they seem to be things that are already merged * 11:27 arturo: [new k8s] cadvisor is running in the metrics namespace now ([[phab:T237643|T237643]]) === 2020-01-02 === * 22:37 bstorm_: Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly. * 22:19 bstorm_: Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups === 2019-12-17 === * 00:14 bstorm_: Fully enabled encryption at rest for toolsbeta kubernetes === 2019-12-16 === * 23:03 bstorm_: updated the kubeadm-config configmap to match the new init file === 2019-12-04 === * 13:02 arturo: drop puppet prefix `toolsbeta-grid-master`, deprecated and no longer in use * 12:50 arturo: drop puppet prefix `toolsbeta-bastion`, deprecated and no longer in use === 2019-12-02 === * 10:38 arturo: create wildcard DNS record for `*.toolsbeta.wmflabs.org` for use by the new k8s cluster * 10:34 arturo: manually scale nginx-ingress deployment to 5 replicas ([[phab:T239405|T239405]]) === 2019-11-25 === * 10:30 arturo: add puppet cert SANs via hiera to toolsbeta-test-k8s-etcd nodes ([[phab:T238655|T238655]]) === 2019-11-21 === * 14:15 arturo: upgrade new k8s cluster to 1.15.6 using kubeadm (plus kubelet) === 2019-11-15 === * 14:46 arturo: stop live-hacks on toolsbeta-test-k8s-haproxy-1 [[phab:T237643|T237643]] === 2019-11-14 === * 10:32 arturo: live-hacking toolsbeta-test-k8s-haproxy-1 to point to just the k8s apiserver in control-1 Turn on --v=10 in control-1 for extended debug === 2019-11-08 === * 19:36 bstorm_: rebooted the proxy server just in case that fixes something. * 11:58 arturo: adding `profile::toolforge::bastion::nproc: 100` to puppet prefix `toolsbeta-sgebastion` ([[phab:T236202|T236202]]) * 11:38 arturo: new k8s: refresh deployment for nginx-ingress with latest changes from puppet === 2019-11-07 === * 21:55 bstorm_: killed pods for ingress admission controller to upgrade to new image [[phab:T215531|T215531]] === 2019-11-06 === * 22:39 bstorm_: upgraded repo version of toollabs-webservice in toolsbeta-stretch to 0.49 -- changes for the new k8s cluster [[phab:T215531|T215531]] * 19:09 bstorm_: added profile::toolforge::proxies in global hiera to try and figure out why it won't let anything use redis [[phab:T237443|T237443]] * 18:53 bstorm_: launching toolsbeta-proxy-2 on a hunch that the config doesn't work well as a standalone [[phab:T237443|T237443]] * 18:46 bstorm_: rebooting toolsbeta-proxy-1 trying to convince redis it is not a read replica [[phab:T237443|T237443]] * 18:29 bstorm_: stopped broken kube-proxy service on toolsbeta-proxy-1 (should probably be puppetized) * 17:35 bstorm_: changing some hiera to work with new proxy host * 12:44 arturo: created VM toolsbeta-proxy-1 ([[phab:T237443|T237443]]) === 2019-11-05 === * 22:50 bstorm_: deployed the new maintain-kubeusers to toolsbeta [[phab:T215531|T215531]] [[phab:T228499|T228499]] === 2019-10-25 === * 23:41 bstorm_: Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster [[phab:T215531|T215531]] [[phab:T215678|T215678]] [[phab:T234231|T234231]] * 16:15 bstorm_: rebooting toolsbeta-test-k8s-worker-1 and -2 === 2019-10-23 === * 12:04 arturo: created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` [[phab:T236074|T236074]] * 11:56 arturo: point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` ([[phab:T236074|T236074]]) * 11:20 arturo: re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile ([[phab:T236074|T236074]]) * 11:10 arturo: re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 ([[phab:T236074|T236074]]) === 2019-10-22 === * 17:43 arturo: re-create VM `toolsbeta-test-k8s-control-1` [[phab:T236074|T236074]] * 15:48 arturo: point DNS record `k8s.toolsbeta.eqiad1.wikimedia.cloud` to the first controller node for the bootstrap [[phab:T236074|T236074]] * 15:30 arturo: created puppet prefix `toolsbeta-test-k8s-control` and delete `toolsbeta-test-k8s-master` [[phab:T236074|T236074]] * 12:27 arturo: refreshed puppet prefix `toolsbeta-test-k8s-control` with latest info [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=12:26 arturo: created 3 VMs `toolsbeta-test-k8s-control-{1,2,3}` T236074}} * 12:15 arturo: refresh IP addr of FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` [[phab:T236074|T236074]] * 12:14 arturo: delete FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=11:57 arturo: created 2 new VMS `toolsbeta-test-k8s-haproxy-{1,2}` T236074}} * 11:54 arturo: created puppet prefix `toolsbeta-test-k8s-haproxy` and delete `toolsbeta-test-k8s-lb` [[phab:T236074|T236074]] === 2019-10-21 === * 15:13 arturo: refresh config in prefix puppet `toolsbeta-test-k8s-etcd` to account for new servers [[phab:T236074|T236074]] * {{safesubst:SAL entry|1=15:07 arturo: create 3 VMs toolsbeta-test-k8s-etcd-{1,2,3} T236074}} * 14:58 arturo: deleting all toolsbeta-test-* VMs (master, worker, etcd, lb) [[phab:T236074|T236074]] === 2019-10-18 === * 16:33 arturo: created DNS zone `toolsbeta.eqiad1.wikimedia.cloud` * 09:06 arturo: remove puppet prefix toolsbeta-valhallasw-puppet-compiler (unused) * {{safesubst:SAL entry|1=09:00 arturo: remove puppet prefix toolsbeta-arturo-k8s-{etcd,master,worker} (unused)}} * {{safesubst:SAL entry|1=08:59 arturo: refresh role for servers in toolsbeta-test-k8s-{master,worker}}} * 08:58 arturo: remove puppet prefix etcd-k8s-ctest (unused) === 2019-10-14 === * 12:26 arturo: delete VM `toolsbeta-test-proxy-01` no longer required * 12:26 arturo: created security group arturo-test-dynamicproxy-backend to tests stuff related to [[phab:T234037|T234037]] === 2019-10-09 === * 11:59 arturo: re-create toolsbeta-test-proxy-01 as Debian Buster ([[phab:T235059|T235059]]) === 2019-10-08 === * 14:14 arturo: created puppet prefix `toolsbeta-test-proxy` for testing stuff related to [[phab:T234037|T234037]] * 12:27 arturo: created VM toolsbeta-test-proxy-01 for testing stuff related to [[phab:T234037|T234037]] === 2019-10-07 === * 19:12 Krenair: reboot toolsbeta-sgecron-01 toolsbeta-sgewebgrid-generic-0901 toolsbeta-sgewebgrid-lighttpd-0901 due to nfs stale issue === 2019-09-25 === * 23:31 bd808: Updated user list for "roots" sudoer policy * 23:30 bd808: Granted Krenair projectadmin === 2019-09-05 === * {{safesubst:SAL entry|1=15:08 zhuyifei1999_: `sudo truncate -s 0 /var/log/exim4/paniclog` on toolsbeta-{sgewebgrid-{lighttpd,generic}-0901,sgecron-01}.toolsbeta.eqiad.wmflabs because of email spam}} === 2019-08-12 === * 20:40 phamhi: toolsbeta-test-puppet-sandbox instance created for [[phab:T230147|T230147]] === 2019-08-09 === * 10:51 arturo: rebalance load: reallocating toolsbeta-sgewebgrid-lighttpd-0901 from cloudvirt1018 to cloudvirt1003 === 2019-07-24 === * 20:48 bstorm_: rebuilt toolsbeta-test cluster with the internal version of the pause container [[phab:T228887|T228887]] [[phab:T215531|T215531]] * 19:02 bstorm_: doing a clean rebuild of the toolsbeta-test-k8s cluster === 2019-07-18 === * 16:04 arturo: re-create VMs toolsbeta-test-k8s-{master,worker}-* * 12:47 arturo: create toolsbeta-test-k8s-etcd-2 as buster to check status of latest puppet code ([[phab:T226098|T226098]]) * 12:00 arturo: create toolsbeta-test-k8s-worker-2 as buster to check status of latest puppet code * {{safesubst:SAL entry|1=09:28 arturo: re-create toolsbeta-test-k8s-master-{1,2,3} as buster to test T228267}} === 2019-07-17 === * 09:51 arturo: re-create VM toolsbeta-test-k8s-worker-1 as Debian Buster [[phab:T215531|T215531]] * 09:13 arturo: create VM toolsbeta-test-k8s-master-4 (Debian Buster) [[phab:T215531|T215531]] === 2019-07-15 === * 12:29 arturo: create `toolsbeta-test-k8s-etcd` puppet prefix * 12:27 arturo: create `toolsbeta-test-k8s-etcd-1` VM [[phab:T215531|T215531]] === 2019-07-03 === * 10:49 arturo: recreate `toolsbeta-test-k8s-master-1` VM ([[phab:T215531|T215531]]) * 09:32 arturo: create `toolsbeta-test-k8s-worker-1` VM and a puppet prefix for it ([[phab:T215531|T215531]]) * 09:22 arturo: delete all `toolsbeta-arturo-k8s-*` instances. We no longer require them per new approach at [[phab:T215531|T215531]] === 2019-07-02 === * 17:24 arturo: `aborrero@toolsbeta-test-k8s-lb-01:~ $ sudo generate_haproxy_default.sh` ([[phab:T215531|T215531]]) * 10:32 arturo: re-creating toolsbeta-test-k8s-master-1 ([[phab:T215531|T215531]]) for it to be created without swap === 2019-07-01 === * 17:13 arturo: re-creating instance `toolsbeta-test-k8s-master-1` with more CPU for [[phab:T215531|T215531]] * 17:03 arturo: updated FQDN `toolsbeta-k8s-master.toolsbeta.wmflabs.org` with 172.16.6.9 (the new LB VM) for [[phab:T215531|T215531]] * 17:02 arturo: re-creating instance `toolsbeta-test-k8s-lb-01` with more CPU for [[phab:T215531|T215531]] * 16:58 arturo: add puppet prefix `toolsbeta-test-k8s-lb` for [[phab:T215531|T215531]] * 11:50 arturo: add sssd hiera config for `toolsbeta-test-k8s-master` prefix === 2019-06-28 === * 19:10 bstorm_: [[phab:T215531|T215531]] removed toolsbeta-arturo-k8s-master-2/3 and added toolsbeta-test-k8s-master-1 for testing kubeadm === 2019-06-25 === * 10:35 arturo: create puppet prefix `toolsbeta-arturo-k8s-worker` for [[phab:T215531|T215531]] * 10:35 arturo: create 2 VMs toolsbeta-arturo-k8s-worker-[1,2] for [[phab:T215531|T215531]] === 2019-06-21 === * 11:42 arturo: re-create 3 VMs toolsbeta-arturo-k8s-etcd-[1-3] to test latest puppet code in [[phab:T226098|T226098]] === 2019-06-19 === * 10:39 arturo: add myself to the `toolsbeta.admin` LDAP group ([[phab:T225303|T225303]]) === 2019-06-14 === * 16:24 bstorm_: Manually failed "back" to the toolsbeta-sgegrid-master to get the grid functioning again in toolsbeta * 16:03 bstorm_: [[phab:T221721|T221721]] hard rebooted toolsbeta-sgegrid-master because it had oomkilled basically everything * 15:55 bstorm_: [[phab:T221721|T221721]] deleted toolsbeta-proxy-01 until it can be actively worked on. * 15:51 bstorm_: deleted toolsbeta-k8s-lb-01 since it isn't being actively worked on just now === 2019-06-06 === * 12:14 arturo: [[phab:T215531|T215531]] create 3 VMs `toolsbeta-arturo-k8s-etcd-[1-3]` * 12:13 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-etcd`* puppet prefix * 12:12 arturo: [[phab:T215531|T215531]] add `toolsbeta-arturo-k8s-test` puppet prefix === 2019-06-05 === * 12:40 arturo: rebase git repos in toolsbeta-puppetmaster-02. There was some rebase problems in labs/private that required me re-creating by hand one of the [local] patches (puppetdb secrets) * 12:33 arturo: drop VM instances toolsbeta-k8s-master-arturo-[1-3] and create toolsbeta-arturo-k8s-master-[1-3] [[phab:T215531|T215531]] * 12:32 arturo: drop puppet prefix `toolsbeta-k8s-master-arturo` and create `toolsbeta-arturo-k8s-master` since there is also `toolsbeta-k8s-master` which get applied to my VMs [[phab:T215531|T215531]] * 11:42 arturo: create VM `toolsbeta-k8s-master-arturo-3` for [[phab:T215531|T215531]] (so I have 3 master nodes in this k8s deployment) * 11:38 arturo: delete instances arturo-sgeexec-sssd-test-2, arturo-sgeexec-sssd-test-1, arturo-bastion-sssd-test, unused === 2019-05-24 === * 11:49 arturo: [[phab:T224273|T224273]] create `toolsbeta-k8s-master-arturo` puppet prefix in horizon * 11:45 arturo: [[phab:T224273|T224273]] create toolsbeta-k8s-master-arturo-[12] stretch VMs * 11:17 arturo: install by hand some openstack client packages that puppet would refuse to install in toolsbeta-k8s-master-01 * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc in toolsbeta-k8s-master-01: * 11:12 arturo: mangle sources.list to handle some apt warnings related to missing repos, etc === 2019-05-07 === * 10:22 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-exec` puppet prefix * 10:20 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-generic` puppet prefix * 10:19 arturo: [[phab:T219362|T219362]] drop the `toolsbeta-webgrid-lighttpd` puppet prefix === 2019-04-25 === * 04:17 andrewbogott: edited resolv.conf on unpuppetized instances to use the new nameserver: toolsbeta-docker-registry-01, toolsbeta-k8s-lb-01, toolsbeta-proxy-01, toolsbeta-puppetdb-01, toolsbeta-sgegrid-master === 2019-04-12 === * 23:34 mutante: - toolsbeta-k8s-master-01 - was out of disk space on / , puppet failed to run because out of disk, rename existing syslog.1.gz, gzip syslog.1, rename existing daemon.log.1.gz, gzip daemong.log.1 * 00:05 andrewbogott: migrating remaining VMs to eqiad1-r === 2019-03-25 === * 18:00 bd808: All Trusty instances shutdown and now in process of deleting * 17:42 bd808: Preparing to shutdown beta Trusty job grid === 2019-03-22 === * 13:59 arturo: create VMs arturo-sgeexec-sssd-test-[12] for testing [[phab:T218126|T218126]] === 2019-03-15 === * 10:23 arturo: create VM `arturo-bastion-sssd-test` ([[phab:T218126|T218126]]) === 2019-02-20 === * 14:58 andrewbogott: moving toolsbeta-grid-master and toolsbeta-puppetmaster-02 to labvirt1003 === 2019-02-14 === * 18:30 andrewbogott: moving toolsbeta-puppetdb-01 to labvirt1002 === 2018-12-04 === * 18:43 arturo: some hiera keys reallocated, see https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/477607/ === 2018-11-26 === * 13:26 arturo: [[phab:T210098|T210098]] VM=toolsbeta-sgebastion-03 * 13:25 arturo: [[phab:T210098|T210098]] install systemd239 from stretch-backports and restart VM === 2018-11-08 === * 10:01 arturo: make myself projectadmin to test toolforge stuff on stretch (specifically [[phab:T207970|T207970]]) === 2018-10-22 === * 21:20 bstorm_: launched a stretch/sonofgridengine master server === 2018-09-19 === * 20:11 bstorm_: toolsbeta-puppetmaster-02 is now the puppetmaster and puppetdb works for toolsbeta -- [[phab:T200557|T200557]] * 17:24 bstorm_: new puppetmaster is toolsbeta-puppetmaster-02, however, manual changes are required on each client, so it will be broken for a bit (enabling puppetdb for [[phab:T200557|T200557]]) * 17:06 bstorm_: working on replacing puppetmaster with one running stretch, as part of adding puppetdb === 2018-07-22 === * 14:28 zhuyifei1999_: backed up Neha16's changes to toolsbeta-bastion-01:/usr/lib/python2.7/dist-packages/toollabs to toollabs.bak in the same dir via cp -a, and re-install webservice code on the bastion to debug [[phab:T156626|T156626]] === 2018-07-18 === * 10:46 harej: Deleted toolsbeta-flynn-01 === 2018-07-12 === * 23:06 bstorm_: Got the grid master running === 2018-06-28 === * 16:34 chicocvenancio: adding harej as root for flynn testing === 2018-06-27 === * 22:35 chicocvenancio: add harej as project admin to test Flynn stuff === 2018-06-22 === * 22:26 chicocvenancio: reconfigured toolsbeta-paws-master-01 kubelet to test image pruning * 09:39 zhuyifei1999_: fixed that by running `sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl.bak` then following the red instructions * 09:33 zhuyifei1999_: puppet is broken on toolsbeta-bastion-01, investigating * 09:03 zhuyifei1999_: killing and rebuilding toolsbeta-bastion-01 * 08:31 zhuyifei1999_: on toolsbeta-bastion-01, killed /etc/apt/sources.list.d/jonathonf-python-2_7-trusty.list ppa, downgraded python from 2.7.14 to 2.7.5, and reinstalled toollabs-webservice * 07:56 andrewbogott: someone removed /usr/bin/webservice === 2018-05-15 === * 07:26 zhuyifei1999_: applied {{Gerrit|5324236}} via toolsbeta-puppetmaster-01 [[phab:T190893|T190893]] * 05:28 zhuyifei1999_: Making project puppetmaster at toolsbeta-puppetmaster-01 === 2018-05-08 === * 02:18 zhuyifei1999_: manually created flannel etcd key [[phab:T190893|T190893]] === 2018-05-07 === * 19:01 zhuyifei1999_: install kubernetes-client on toolsbeta-worker-1001 to debug stuffs * 18:41 zhuyifei1999_: rebuilding toolsbeta-k8s-etcd-01 * 17:58 zhuyifei1999_: cleanup from maintain-kubeusers using the wrong project to create tool home dirs: `find /data/project/ -mindepth 1 -maxdepth 1 -type d \! -user 0 {{!}} (while read dir; do id toolsbeta.`basename $dir` 2> /dev/null {{!}}{{!}} sudo rm -rfv $dir; done)` * 16:41 zhuyifei1999_: rebuild toolsbeta-k8s-master-01 because I can't figure out why puppet can't update maintain-kubeusers.systemd === 2018-05-06 === * 04:06 zhuyifei1999_: locally patched `/usr/lib/python2.7/dist-packages/toollabs/common/tool.py` on bastion and webgrid-lighttpd === 2018-05-05 === * 19:51 zhuyifei1999_: `systemctl mask maintain-kubeusers` because it's causing a mess, tries to get the tool list from toolforge [[phab:T190893|T190893]] * 18:40 zhuyifei1999_: to unblock k8s testing while waiting on https://gerrit.wikimedia.org/r/430539, installed the package directly on `toolsbeta-k8s-master-01` with `$ sudo apt install python3-yaml` === 2018-05-02 === * 21:02 zhuyifei1999_: copy over labs/private:/hieradata/labs/tools/common.yaml to project puppet hiera * 20:37 bd808: Added Neha16 as a project admin for work on [[phab:T175768|T175768]] * 20:31 zhuyifei1999_: nuke webservice instances and rebuild * 20:31 zhuyifei1999_: Added k8s_infrastructure_users to project hiera on horizon [[phab:T192618|T192618]] === 2018-04-20 === * 00:20 zhuyifei1999_: deleted all instances I just created except k8s master because of chicken-and-egg problem === 2018-04-19 === * 22:10 zhuyifei1999_: the trusty instances ask me for my password. the jessie instances don't like my ssh key. :( * 21:59 zhuyifei1999_: got 'Error: RecordSet belongs in a child zone: toolsbeta.wmflabs.org', using tools-beta.wmflabs.org instead * 21:57 zhuyifei1999_: Add proxy toolsbeta.wmflabs.org => toolsbeta-proxy-01.toolsbeta.eqiad.wmflabs * 21:43 zhuyifei1999_: Start creating instances for webservice setup [[phab:T190893|T190893]] === 2018-03-30 === * 22:40 zhuyifei1999_: copied over many prefix puppet configuration in horizon from toolforge [[phab:T190893|T190893]] === 2018-03-14 === * 18:07 chicocvenancio: updated paws-beta k8s cluster and nodes to v1.9.4 for [[phab:T189680|T189680]] === 2018-03-05 === * 19:33 chicocvenancio: added Zhuyifei1999 as project admin === 2018-02-09 === * 01:11 bd808: Removed Yuvipanda at user request ([[phab:T186289|T186289]]) === 2017-08-07 === * 14:09 andrewbogott: deleted etcd-k8s-CTEST and k8s-master-CTEST === 2017-04-26 === * 15:38 madhuvishy: add Madhuvishy as projectadmin === 2016-10-07 === * 19:30 valhallasw`cloud: (puppet certs, to be precise) * 19:30 valhallasw`cloud: fixed certs on toolsbeta-vagrant3-scfc.toolsbeta.eqiad.wmflabs === 2016-10-04 === * 19:31 valhallasw`cloud: puppet is broken due to incorrect certificates. Cleaning up ('puppet cert clean toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs' on puppetmaster3, 'rm -f /var/lib/puppet/client/ssl/certs/toolsbeta-webgrid-lighttpd-1406.toolsbeta.eqiad.wmflabs.pem' on host, for all hosts that I got emails for) === 2016-09-08 === * 17:11 bd808: Added BryanDavis (self) to project as admin === 2016-08-29 === * 19:20 yuvipanda: reboot toolsbeta-master, seems, uh, stuck * 19:18 yuvipanda: reboot toolsbeta-mail, seems, uh, stuck * 18:48 yuvipanda: reboot toolsbeta-puppetmaster3, puppet run process became Zommmmbiiiieeee, ate all my brains === 2016-07-03 === * 15:02 yuvipanda: migrating toolsbeta-valhallasw-puppet-compiler to labvirt1011 to ease pressure on labvirt1010 === 2016-05-27 === * 18:57 valhallasw`cloud: sudo qconf -Ae /var/lib/gridengine/etc/exechosts/toolsbeta-exec-1209.toolsbeta.eqiad.wmflabs === 2016-05-26 === * 15:08 valhallasw`cloud: toolsbeta-mail has high load (1.0) without clear origin, so rebooting the host === 2015-10-13 === * 19:21 valhallasw`cloud: started building toolsbeta-bastion. === 2015-09-07 === * 18:50 valhallasw`cloud: role::bastion is now applied on -exec-101. Now for the package_builder manifest... * 18:30 valhallasw`cloud: applied role::toollabs::bastion on toolsbeta-exec-101 (spinning up a whole new instance will take ages) === July 4 === * 12:57 valhallasw`cloud: restarting toolsbeta-webproxy, no response on port 22 === July 2 === * 14:55 valhallasw`cloud: toolsbeta-webproxy does not respond at all to SSH; rebooting === July 1 === * 19:47 valhallasw`cloud: still can't login :/ not sure if this is a remainder of the NFS failure or something else; maybe a puppet run will solve it? * 19:44 valhallasw`cloud: restarting toolsbeta-exec-01 and toolsbeta-mail as I can't login === June 7 === * 14:44 valhallasw: updated /var/lib/git/operations/puppet to make sure the other hosts get the memo * 14:42 YuviPanda: run sudo sed -i 's/GlobalSign_CA.pem/ca-certificates.crt/' /etc/ldap/ldap.conf on toolsbeta-puppetmaster3 to fix broken LDAP TLS config === May 11 === * 18:14 valhallasw: building toolsbeta-pbuilder to experiment with pbuilder for building packages === May 2 === * 11:11 valhallasw`cloud: commenting out include ::elasticsearch::ganglia in role::logstash seems to work. I think we have to write our own tools logstash roles anyway in the end, as the role::logstash code contains e.g. mediawiki specific code * 10:37 valhallasw`cloud: that doesn't seem to be applied... setting has_ganglia: false manually in wikitech hiera * 10:30 valhallasw`cloud: pulled new changes into puppetmaster to get https://github.com/wikimedia/operations-puppet/commit/4afd23d8e2905a84ef211ad92e8314173eb743ba in * 10:25 valhallasw`cloud: set Hiera variable "elasticsearch::cluster_name": toolsbeta-logstash-eqiad * 10:09 valhallasw`cloud: created [[Nova_Resource:I-00000c01.eqiad.wmflabs|toolsbeta-logstash]] to play around with logstash and figure out what we need for tools ([[phab:T97861]]) === April 26 === * 18:18 valhallasw`cloud: having some issues with puppet-test, so postponing for now * 17:12 valhallasw`cloud: deploying https://gerrit.wikimedia.org/r/#/c/206118/ on tools-beta using puppet-test === March 31 === * 00:27 andrewbogott: shut down toolsbeta-webgrid-03 to conserve resources. It can be restarted when needed. === September 20 === * 20:09 andrewbogott_afk: moved toolsbeta-exec-01 and toolsbeta-scfc-icinga-test off of virt1006 === July 22 === * 11:36 scfc_de: Removed andrewbogott_afk, Coren, petan, YuviPanda from service group admin to prevent further spamming :-) === August 19 === * 12:44 petan: rebooting apache it seems to be frozen === August 4 === * 23:50 scfc_de: Added scfc_de to local-admin so I don't log myself out again :-) === July 6 === * 19:42 petan: rebooting login === June 26 === * 08:03 wm-bot: petrb: updating logsplitter === June 24 === * 14:47 wm-bot: petrb: rebooting exec-01 to fix the grid weird info * 13:43 scfc_de: Made scfc root. * 13:42 scfc_de: Created toolsbeta-puppetmaster. * 11:09 YuviPanda: Granted yuvipanda root on toolsbeta === June 21 === * 13:46 wm-bot: petrb: rebooting all servers === June 17 === * 08:31 petan: switching all instances to nfs === June 16 === * 15:37 petan: importing sudo policies of tools * 15:36 petan: importing security groups of tools * 15:36 petan: blah {{SAL|Project Name=toolsbeta}} <noinclude>[[Category:SAL]]</noinclude> 6nc681aoki1riwfzz98dmka0gzw4fu1 0 7919 2452782 2452771 2026-08-31T02:00:25Z Stashbot 7414 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image 2452782 wikitext text/x-wiki == 2026-08-31 == * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 7vovtpkyfziqrrps6z172f8y8seal91 2452783 2452782 2026-08-31T02:08:41Z Stashbot 7414 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) 2452783 wikitext text/x-wiki == 2026-08-31 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 87kqb4xqnlif9w2q2r3xher6g7p7msd 2452784 2452783 2026-08-31T03:11:06Z Stashbot 7414 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525|Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] 2452784 wikitext text/x-wiki == 2026-08-31 == * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> bobroermp5t7uw7guwf71lsrm8pan6v 2452786 2452784 2026-08-31T03:29:28Z Stashbot 7414 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525|Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2452786 wikitext text/x-wiki == 2026-08-31 == * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> sh3blgwxqfbgljbrrt1kezdq6zq4prz 2452788 2452786 2026-08-31T03:31:11Z Stashbot 7414 tstarling@deploy1003: tstarling: Continuing with deployment 2452788 wikitext text/x-wiki == 2026-08-31 == * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 15albuis6nbx8rnupr2h4hfpzddjz8v 2452789 2452788 2026-08-31T03:44:26Z Stashbot 7414 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525|Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) 2452789 wikitext text/x-wiki == 2026-08-31 == * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 7n579fbj15fy3m6s57hhay1rfwpyvla 2452791 2452789 2026-08-31T07:04:12Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. 2452791 wikitext text/x-wiki == 2026-08-31 == * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 2pqzusrzm1szonul2w41fb46y3ipbea 2452792 2452791 2026-08-31T07:05:06Z Stashbot 7414 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. 2452792 wikitext text/x-wiki == 2026-08-31 == * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> bx97fm6rf0ep8w8czohobz1mhjq7jvn 2452794 2452792 2026-08-31T07:15:14Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474|kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] 2452794 wikitext text/x-wiki == 2026-08-31 == * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> sr88n5i9t2lbv0vgt042giupqe10wmv 2452795 2452794 2026-08-31T07:19:45Z Stashbot 7414 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474|kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2452795 wikitext text/x-wiki == 2026-08-31 == * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> t8xe4c8kl6kqvken1gzsmiwvooqh8ph 2452796 2452795 2026-08-31T07:20:37Z Stashbot 7414 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply 2452796 wikitext text/x-wiki == 2026-08-31 == * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> b0ccsczjlwhx2b15r819psai6386i5j 2452797 2452796 2026-08-31T07:21:38Z Stashbot 7414 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply 2452797 wikitext text/x-wiki == 2026-08-31 == * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> d30x3kbt71lkuzxpci13e9a2lsjy9vs 2452798 2452797 2026-08-31T07:22:27Z Stashbot 7414 kartik@deploy1003: kartik, revi: Continuing with deployment 2452798 wikitext text/x-wiki == 2026-08-31 == * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> amhps14up4bkotvpw12077xeno6hacj 2452799 2452798 2026-08-31T07:30:15Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474|kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) 2452799 wikitext text/x-wiki == 2026-08-31 == * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 2wq244exl698djwu3sgdi4pukr57wle 2452800 2452799 2026-08-31T07:31:44Z Stashbot 7414 moritzm: installing openssl security updates 2452800 wikitext text/x-wiki == 2026-08-31 == * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> gjpvcb2a4455rdtjungw934gttfr32e 2452801 2452800 2026-08-31T07:33:27Z Stashbot 7414 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181|thwikibooks: update wordmark and tagline (T436426)]] 2452801 wikitext text/x-wiki == 2026-08-31 == * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> kz9339xysx3vi0jcgylcz2z8gfgikti 2452802 2452801 2026-08-31T07:37:22Z Stashbot 7414 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181|thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. 2452802 wikitext text/x-wiki == 2026-08-31 == * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> a4o0yv4ozuoxmbenikqbqzpj9vi4t95 2452803 2452802 2026-08-31T07:38:31Z Stashbot 7414 kartik@deploy1003: hamishz, kartik: Rolling back deployment 2452803 wikitext text/x-wiki == 2026-08-31 == * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 4pzucq0ve2mmex6j5kw8yuexxszeeju 2452804 2452803 2026-08-31T07:41:08Z Stashbot 7414 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181|thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) 2452804 wikitext text/x-wiki == 2026-08-31 == * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 47s9pyi60tmymbjkfkkq9pd2mnvr8gc 2452805 2452804 2026-08-31T08:01:56Z Stashbot 7414 moritzm: installing Linux 6.12.107 on Trixie hosts 2452805 wikitext text/x-wiki == 2026-08-31 == * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ai5qejv6x5y2z01faq30c2s7z3i1wwt 2452806 2452805 2026-08-31T08:39:32Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe 2452806 wikitext text/x-wiki == 2026-08-31 == * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> t2qlalgvq8ihheln1erdnm1ka69ppb6 2452807 2452806 2026-08-31T08:39:55Z Stashbot 7414 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe 2452807 wikitext text/x-wiki == 2026-08-31 == * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 9hayf62ek0l1caoge9gk9e6vccclbjk 2452809 2452807 2026-08-31T08:42:49Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet 2452809 wikitext text/x-wiki == 2026-08-31 == * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> gz25ozxpuqn04h6cpe2pui964os95m6 2452810 2452809 2026-08-31T08:43:54Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe 2452810 wikitext text/x-wiki == 2026-08-31 == * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> os8um55i2c71ptygcouimnyd2bcf0dn 2452811 2452810 2026-08-31T08:49:33Z Stashbot 7414 moritzm: installing emacs security updates 2452811 wikitext text/x-wiki == 2026-08-31 == * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ocm4e6degbehgha1pqce1p73l469e7t 2452812 2452811 2026-08-31T08:56:37Z Stashbot 7414 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe 2452812 wikitext text/x-wiki == 2026-08-31 == * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> kbfdihrpuammdms03g5hv3yvnp25ebq 2452813 2452812 2026-08-31T08:57:07Z Stashbot 7414 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 2452813 wikitext text/x-wiki == 2026-08-31 == * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> in3g20ty3ph5zzuw4pqz2iw0xu1mqz9 2452814 2452813 2026-08-31T08:58:17Z Stashbot 7414 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections T407942 2452814 wikitext text/x-wiki == 2026-08-31 == * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> gvlq8ag2p7ccnnu1g9f2mrc567utpws 2452815 2452814 2026-08-31T08:58:19Z Stashbot 7414 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning 2452815 wikitext text/x-wiki == 2026-08-31 == * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 2abrcb9k2dc4fq4w8tgy35d6dqaeg5h 2452816 2452815 2026-08-31T09:06:35Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet 2452816 wikitext text/x-wiki == 2026-08-31 == * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ogz6214tr2hh0d5d3ri07nsk4m8i6af 2452818 2452816 2026-08-31T09:16:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet 2452818 wikitext text/x-wiki == 2026-08-31 == * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> e09i97mxr5c7yi3xbnf86wn5gfqzff6 2452819 2452818 2026-08-31T09:16:43Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet 2452819 wikitext text/x-wiki == 2026-08-31 == * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> knmhkagdijibn1t130e2ual993w28df 2452820 2452819 2026-08-31T09:21:21Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2452820 wikitext text/x-wiki == 2026-08-31 == * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 3iszskg44gbwnl65pgctvcedwm9czwv 2452821 2452820 2026-08-31T09:24:05Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad 2452821 wikitext text/x-wiki == 2026-08-31 == * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> b7wnog6qwulr64dq9r7uoe49rkiv4bl 2452822 2452821 2026-08-31T09:24:09Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet 2452822 wikitext text/x-wiki == 2026-08-31 == * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 91camrib7wvsosgwpkl5o8vdc59rg30 2452823 2452822 2026-08-31T09:24:09Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet 2452823 wikitext text/x-wiki == 2026-08-31 == * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> gii59x3r5et2ttzhoqnd1bvfocxi575 2452824 2452823 2026-08-31T09:29:11Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL 2452824 wikitext text/x-wiki == 2026-08-31 == * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> aqjli52d4dub965og9g28jl1blxaww6 2452825 2452824 2026-08-31T09:29:12Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet 2452825 wikitext text/x-wiki == 2026-08-31 == * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> sg5x7586rlbtvpdz0ik9hu47urshnzb 2452826 2452825 2026-08-31T09:29:13Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet 2452826 wikitext text/x-wiki == 2026-08-31 == * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> a0lfothxwblgs9nhsa11rndktlnfkzt 2452827 2452826 2026-08-31T09:29:18Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet 2452827 wikitext text/x-wiki == 2026-08-31 == * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 5b8fkp19112b5za82t8apx8x4mcebz0 2452828 2452827 2026-08-31T09:29:18Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet 2452828 wikitext text/x-wiki == 2026-08-31 == * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> bt6nz33mi6io7n7okgchfazeqdx68xx 2452829 2452828 2026-08-31T09:30:15Z Stashbot 7414 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad 2452829 wikitext text/x-wiki == 2026-08-31 == * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> jjhka47y8sixpi4s9gplqile5lq4c8j 2452830 2452829 2026-08-31T09:34:13Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet 2452830 wikitext text/x-wiki == 2026-08-31 == * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 5yxul1x168vjgz4op2tl9mjcjm3h1y4 2452831 2452830 2026-08-31T09:34:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet 2452831 wikitext text/x-wiki == 2026-08-31 == * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> s7hmtrpl14mn0lrcz7drz2f08jdtpwv 2452832 2452831 2026-08-31T09:34:14Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad 2452832 wikitext text/x-wiki == 2026-08-31 == * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> dbehwiuf797vi6mvfchm0za39bji4h0 2452833 2452832 2026-08-31T09:37:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw 2452833 wikitext text/x-wiki == 2026-08-31 == * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> p6s54di00acm9yxkf79l2ltnqtw95fr 2452834 2452833 2026-08-31T09:37:15Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet 2452834 wikitext text/x-wiki == 2026-08-31 == * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> tr4rca8rtdg39tvoqfcv30g82cq5wui 2452835 2452834 2026-08-31T09:37:16Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet 2452835 wikitext text/x-wiki == 2026-08-31 == * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> rgdkj5uj56ntgb2f9h3vubol0u3tm9k 2452836 2452835 2026-08-31T09:42:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet 2452836 wikitext text/x-wiki == 2026-08-31 == * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ie84deono6dv6hgd22pn1idkdstp62g 2452837 2452836 2026-08-31T09:42:05Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet 2452837 wikitext text/x-wiki == 2026-08-31 == * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> qh9tbxyrpsemix90vxhwncucp0upz9x 2452838 2452837 2026-08-31T09:42:11Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet 2452838 wikitext text/x-wiki == 2026-08-31 == * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> s6acrfumjojvy12an2k3b36pf4l7yzt 2452839 2452838 2026-08-31T09:42:12Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet 2452839 wikitext text/x-wiki == 2026-08-31 == * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> dbwhkhps1cq82uxv5x671w0t6xukgwu 2452840 2452839 2026-08-31T09:47:20Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet 2452840 wikitext text/x-wiki == 2026-08-31 == * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 8fxu4opuh3fodlqlpe2eqxs2qvjncpc 2452841 2452840 2026-08-31T09:47:21Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet 2452841 wikitext text/x-wiki == 2026-08-31 == * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 54k6ix8t5wr94xvmhgkput9tpmos4so 2452842 2452841 2026-08-31T09:47:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw 2452842 wikitext text/x-wiki == 2026-08-31 == * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 9gbwuuz1d9qx62sf1sk1bkvwepdn5ue 2452843 2452842 2026-08-31T09:59:40Z Stashbot 7414 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 2452843 wikitext text/x-wiki == 2026-08-31 == * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> s2fhcmr0a5xe8dvjnq1x0hhw7761htn 2452844 2452843 2026-08-31T10:01:02Z Stashbot 7414 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply 2452844 wikitext text/x-wiki == 2026-08-31 == * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 2tw1r7d2odrsgrm9nwse830dijxpc38 2452845 2452844 2026-08-31T10:02:09Z Stashbot 7414 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply 2452845 wikitext text/x-wiki == 2026-08-31 == * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 8iq4tssp94koxxehpswdj5q22sk4vcn 2452846 2452845 2026-08-31T10:07:03Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{aux-k8s-worker[1006-1009].eqiad.wmnet} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2452846 wikitext text/x-wiki == 2026-08-31 == * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 96qi5io4dbd0rf6on2hctax0n7akxl1 2452847 2452846 2026-08-31T10:07:07Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet 2452847 wikitext text/x-wiki == 2026-08-31 == * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 27vn4vkerw5y7c7fzmy2dl4w4pumh80 2452848 2452847 2026-08-31T10:07:29Z Stashbot 7414 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 (T434766) 2452848 wikitext text/x-wiki == 2026-08-31 == * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> q6psgrrmzxaxnspbamlag4fsoyli07q 2452849 2452848 2026-08-31T10:07:43Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet 2452849 wikitext text/x-wiki == 2026-08-31 == * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> kzttic6fmzbh2gfor235c5que6y24cx 2452850 2452849 2026-08-31T10:10:16Z Stashbot 7414 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply 2452850 wikitext text/x-wiki == 2026-08-31 == * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> gmm08fv70ayzsu8ijsov3d7ltpimsq5 2452851 2452850 2026-08-31T10:12:24Z Stashbot 7414 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply 2452851 wikitext text/x-wiki == 2026-08-31 == * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> t1w6z8h0v7xto98zjdc6dg904zctvfq 2452852 2452851 2026-08-31T10:12:51Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet 2452852 wikitext text/x-wiki == 2026-08-31 == * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> h6ifh58qby32cmu7cnrk127itgki73l 2452853 2452852 2026-08-31T10:12:51Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet 2452853 wikitext text/x-wiki == 2026-08-31 == * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> efowge16fzu92d1bwq3my91ix2jccfl 2452854 2452853 2026-08-31T10:12:58Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet 2452854 wikitext text/x-wiki == 2026-08-31 == * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 315s42yh05wnzzuuvt7dphvcbv5lqnb 2452855 2452854 2026-08-31T10:13:33Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet 2452855 wikitext text/x-wiki == 2026-08-31 == * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> n9fi3d480xysmcfl0mkggs4xcxcrs4u 2452856 2452855 2026-08-31T10:14:07Z Stashbot 7414 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply 2452856 wikitext text/x-wiki == 2026-08-31 == * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ehju7njbuz9wvstmjmw7b0h8gazgwur 2452857 2452856 2026-08-31T10:15:53Z Stashbot 7414 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply 2452857 wikitext text/x-wiki == 2026-08-31 == * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> cnzksaridf0t2tylm5q8wrsbxrgwxv3 2452859 2452857 2026-08-31T10:18:42Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet 2452859 wikitext text/x-wiki == 2026-08-31 == * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 80xv6o009ggv5ehmot37ikl8bmrlki6 2452860 2452859 2026-08-31T10:18:43Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet 2452860 wikitext text/x-wiki == 2026-08-31 == * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> n6ju5l9415zv21vb49zfscd9101tc4t 2452861 2452860 2026-08-31T10:18:48Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet 2452861 wikitext text/x-wiki == 2026-08-31 == * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> ji9z4ojhgbi42n424z99ijpn58l91ax 2452862 2452861 2026-08-31T10:19:24Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet 2452862 wikitext text/x-wiki == 2026-08-31 == * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> lkjtwdcdis9dzq8b5ih6zzz8aele3ed 2452863 2452862 2026-08-31T10:24:41Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet 2452863 wikitext text/x-wiki == 2026-08-31 == * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> rcq6ar0rvt7jxgoq7znczmrg8jtk4ec 2452864 2452863 2026-08-31T10:24:41Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet 2452864 wikitext text/x-wiki == 2026-08-31 == * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> lcmngqe2x2ssnuxj1f7kspcmkhbql8b 2452865 2452864 2026-08-31T10:24:47Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet 2452865 wikitext text/x-wiki == 2026-08-31 == * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> bv74e8s7bitnie4su60eqkgc13akbnz 2452867 2452865 2026-08-31T10:25:22Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet 2452867 wikitext text/x-wiki == 2026-08-31 == * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 7v1mm1w38xq6gmw9lnztxyvdw8xpl1k 2452868 2452867 2026-08-31T10:30:34Z Stashbot 7414 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet 2452868 wikitext text/x-wiki == 2026-08-31 == * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 17sut4nyp0f7sinhclt0g4m4m3q2ej5 2452869 2452868 2026-08-31T10:30:35Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet 2452869 wikitext text/x-wiki == 2026-08-31 == * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 41zrl20xt2natj6fifi48j1odvd5xl2 2452870 2452869 2026-08-31T10:30:36Z Stashbot 7414 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{aux-k8s-worker[1006-1009].eqiad.wmnet} and (A:aux-master-eqiad or A:aux-worker-eqiad) 2452870 wikitext text/x-wiki == 2026-08-31 == * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> r0n84cb57aoamrsai0fgl0zqndm41z6 2452873 2452870 2026-08-31T10:54:06Z Stashbot 7414 kamila@deploy1003: Started scap sync-world: rebuild for T436488 2452873 wikitext text/x-wiki == 2026-08-31 == * 10:54 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T436488|T436488]] * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> cu6qgdde8ds7v2ox4xn76472hi9z28z 2452874 2452873 2026-08-31T11:11:56Z Stashbot 7414 moritzm: powercycle netmon2002, unresponsive 2452874 wikitext text/x-wiki == 2026-08-31 == * 11:11 moritzm: powercycle netmon2002, unresponsive * 10:54 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T436488|T436488]] * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 9ibg0g738kz74lp9uq5jir95usk3cal 2452875 2452874 2026-08-31T11:20:52Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . 2452875 wikitext text/x-wiki == 2026-08-31 == * 11:20 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:11 moritzm: powercycle netmon2002, unresponsive * 10:54 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T436488|T436488]] * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> 22bpkn9fxwp2n4xbgxomye4n7fk8mdq 2452876 2452875 2026-08-31T11:21:09Z Stashbot 7414 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . 2452876 wikitext text/x-wiki == 2026-08-31 == * 11:21 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:20 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:11 moritzm: powercycle netmon2002, unresponsive * 10:54 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T436488|T436488]] * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> q8cjd72fx6ryomx5oh04ugtnyxhde72 2452877 2452876 2026-08-31T11:28:58Z Stashbot 7414 kamila@deploy1003: Finished scap sync-world: rebuild for T436488 (duration: 34m 43s) 2452877 wikitext text/x-wiki == 2026-08-31 == * 11:28 kamila@deploy1003: Finished scap sync-world: rebuild for [[phab:T436488|T436488]] (duration: 34m 43s) * 11:21 ozge@deploy1003: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:20 ozge@deploy1003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'experimental' for release 'main' . * 11:11 moritzm: powercycle netmon2002, unresponsive * 10:54 kamila@deploy1003: Started scap sync-world: rebuild for [[phab:T436488|T436488]] * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:30 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1009.eqiad.wmnet * 10:30 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1009.eqiad.wmnet * 10:25 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1009.eqiad.wmnet * 10:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1008.eqiad.wmnet * 10:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1008.eqiad.wmnet * 10:19 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1008.eqiad.wmnet * 10:18 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1007.eqiad.wmnet * 10:18 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1007.eqiad.wmnet * 10:15 ozge@deploy1003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply * 10:14 ozge@deploy1003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply * 10:13 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1007.eqiad.wmnet * 10:12 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-worker1006.eqiad.wmnet * 10:12 ozge@deploy1003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply * 10:10 ozge@deploy1003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply * 10:07 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 fabfur: enable puppet on A:cp to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314 ([[phab:T434766|T434766]]) * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-worker1006.eqiad.wmnet * 10:07 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P<nowiki>{</nowiki>aux-k8s-worker[1006-1009].eqiad.wmnet<nowiki>}</nowiki> and (A:aux-master-eqiad or A:aux-worker-eqiad) * 10:02 ozge@deploy1003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply * 10:01 ozge@deploy1003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply * 09:59 fabfur: disabled puppet on A:cp to gradually apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1330314, starting from cp6016 * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-codfw * 09:47 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:47 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2003.codfw.wmnet * 09:42 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:42 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl2002.codfw.wmnet * 09:37 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-codfw * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:aux-master-eqiad * 09:34 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:34 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:30 elukey@puppetserver1001: conftool action : set/pooled=true; selector: dnsdisc=pki,name=eqiad * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1003.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:29 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:24 elukey@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host aux-k8s-ctrl1002.eqiad.wmnet * 09:24 elukey@cumin1003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:aux-master-eqiad * 09:21 elukey@cumin1003: START - Cookbook sre.hosts.provision for host pki1002.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts pki1002.eqiad.wmnet * 09:16 elukey@cumin1003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host pki1002.eqiad.wmnet * 09:06 elukey@cumin1003: START - Cookbook sre.hosts.reboot-single for host pki1002.eqiad.wmnet * 08:58 marostegui@cumin1003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 22 hosts with reason: Cloning * 08:58 marostegui: Stop mariadb on sanitarium s2,s4,s6,s7 there will be lag on wikireplicas for those sections [[phab:T407942|T407942]] * 08:57 hashar: Upgrading CI Jenkins 2.555.3 to 2.568.2 * 08:56 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe * 08:49 moritzm: installing emacs security updates * 08:43 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe * 08:42 elukey@cumin1003: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts pki1002.eqiad.wmnet * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe * 08:39 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe * 08:01 moritzm: installing Linux 6.12.107 on Trixie hosts * 07:41 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] (duration: 07m 41s) * 07:38 kartik@deploy1003: hamishz, kartik: Rolling back deployment * 07:37 kartik@deploy1003: hamishz, kartik: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:33 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332181{{!}}thwikibooks: update wordmark and tagline (T436426)]] * 07:31 moritzm: installing openssl security updates * 07:30 kartik@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] (duration: 15m 00s) * 07:22 kartik@deploy1003: kartik, revi: Continuing with deployment * 07:21 jelto@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply * 07:20 jelto@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply * 07:19 kartik@deploy1003: kartik, revi: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 07:15 kartik@deploy1003: Started scap sync-world: Backport for [[gerrit:1332474{{!}}kowiki: convert extendedconfirmed calculation to begin from first edit (T436449)]] * 07:05 brouberol@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. * 07:04 brouberol@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. * 03:44 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] (duration: 33m 20s) * 03:31 tstarling@deploy1003: tstarling: Continuing with deployment * 03:29 tstarling@deploy1003: tstarling: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 03:11 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1332525{{!}}Produnto: use wgCopyUploadProxy to contact GitLab (T421436)]] * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 08m 15s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-30 == * 02:07 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-29 == * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image == 2026-08-28 == * 21:16 ryankemper: [[phab:T415073|T415073]] admin_ng namespace-certificates applied on `staging-codfw`, `staging-eqiad`, `codfw`, `eqiad`; `wikidata-query-gui` cert reissued without `query-legacy-full.wikidata.org` SAN, verified on the wire in both DCs * 21:08 ryankemper@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. * 21:07 ryankemper@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. * 21:06 ryankemper@deploy1003: helmfile [staging-codfw] START helmfile.d/admin 'apply'. * 21:05 ryankemper: [[phab:T415073|T415073]] beginning admin_ng namespace-certificates rollout (gerrit {{Gerrit|1278562}}, drop query-legacy-full.wikidata.org SAN from wikidata-query-gui cert), starting with staging * 20:05 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 20:04 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:47 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:45 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:44 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:43 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:42 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:36 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:35 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:34 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 19:33 lerickson@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply * 19:31 lerickson@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply * 18:33 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 18:13 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 18:09 eevans@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage * 17:53 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:52 eevans@cumin1003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL * 17:50 eevans@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm * 17:32 eevans@cumin1003: START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm * 16:38 sukhe@cumin1003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie * 16:03 sukhe: sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage * 15:43 elukey: powercycle pki1002 after CPU-related failures (host completely unresponsive) - [[phab:T434268|T434268]] * 15:40 elukey@puppetserver1001: conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 15:03 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 14:59 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:54 sukhe@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage * 14:22 sukhe@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 14:22 sukhe@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 14:03 bking@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. * 14:01 bking@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet * 12:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie * 12:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 12:07 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage * 11:56 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie * 11:43 moritzm: add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org [[phab:T416709|T416709]] * 11:43 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:43 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:42 jmm@cumin2003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors * 11:41 jmm@cumin2003: START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors * 11:40 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 11:39 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:38 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cumin1004.eqiad.wmnet - jmm@cumin2003" * 11:23 jmm@cumin2003: START - Cookbook sre.dns.netbox * 11:23 jmm@cumin2003: START - Cookbook sre.ganeti.makevm for new host cumin1004.eqiad.wmnet * 11:12 aokoth@cumin1003: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 10:43 aokoth@cumin1003: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - [[phab:T436069|T436069]] * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cumin2002.codfw.wmnet * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 08:59 jmm@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:58 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cumin2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003" * 08:28 jmm@cumin2003: START - Cookbook sre.dns.netbox * 07:49 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts cumin2002.codfw.wmnet * 06:30 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org * 06:26 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org * 04:34 ryankemper: [[phab:T435862|T435862]] Completed the rolling restart of all 15 production Presto workers; verified `true/true` spill settings across coordinators and workers, full runtime membership, the formerly failing query, and Superset dashboard 757. This is done * 04:31 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:59 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto cluster: Roll restart of all Presto's jvm daemons. * 03:58 ryankemper: [[phab:T435862|T435862]] Restarted both production Presto coordinators (standby first, then active) after enabling JOIN spilling; the active coordinator reports `spill_enabled=true` and `join_spill_enabled=true` and all 15 workers rejoined * 03:42 ryankemper: [[phab:T435862|T435862]] Verified `spill_enabled=true` and `join_spill_enabled=true` on the Presto test cluster after restarting its coordinator and worker; both nodes are active and distributed query `20260828_033922_00002_u9k78` succeeded * 03:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.presto.roll-restart-workers (exit_code=0) for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:38 ryankemper@cumin2003: START - Cookbook sre.presto.roll-restart-workers for Presto an-presto-test cluster: Roll restart of all Presto's jvm daemons. * 03:10 tstarling@deploy1003: Finished scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] (duration: 12m 44s) * 03:05 tstarling@deploy1003: tstarling: Continuing with deployment * 03:02 tstarling@deploy1003: tstarling: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 02:58 tstarling@deploy1003: Started scap sync-world: Backport for [[gerrit:1328752{{!}}Grant produnto-update to groups that have editprotected (T421436)]], [[gerrit:1329728{{!}}Produnto: Add IPv6 range for GitLab (T421436)]] * 02:25 ryankemper: `ryankemper@pcc-db1002:~$ sudo -u jenkins-deploy puppetdb-populate --host an-test-coord1002.eqiad.wmnet` (refreshed stale catalog) * 02:08 mwpresync@deploy1003: Finished scap build-images: Publishing wmf/next image (duration: 07m 53s) * 02:00 mwpresync@deploy1003: Started scap build-images: Publishing wmf/next image * 00:55 ladsgroup@deploy1003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply * 00:54 ladsgroup@deploy1003: helmfile [eqiad] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:53 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:52 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:40 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:39 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply * 00:27 ladsgroup@deploy1003: helmfile [codfw] START helmfile.d/services/thumbor: apply * 00:24 ladsgroup@deploy1003: helmfile [staging] DONE helmfile.d/services/thumbor: apply * 00:23 ladsgroup@deploy1003: helmfile [staging] START helmfile.d/services/thumbor: apply == 2026-08-27 == * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1261.eqiad.wmnet * 23:14 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1261.eqiad.wmnet * 23:07 jasmine_: homer lsw1-c6-eqiad* commit '[[phab:T421711|T421711]]' * 23:05 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:44 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:40 jasmine@cumin1003: START - Cookbook sre.hosts.downtime for 2:00:00 on wikikube-worker1261.eqiad.wmnet with reason: host reimage * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wikikube-worker1261 * 22:20 jasmine@cumin1003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wikikube-worker1261 * 22:19 jasmine@cumin1003: START - Cookbook sre.network.configure-switch-interfaces for host wikikube-worker1261 * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: START - Cookbook sre.dns.wipe-cache wikikube-worker1261.eqiad.wmnet 71.32.64.10.in-addr.arpa 1.7.0.0.2.3.0.0.4.6.0.0.0.1.0.0.3.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0) * 22:19 jasmine@cumin1003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:19 jasmine@cumin1003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wikikube-worker1261 - jasmine@cumin1003" * 22:13 jasmine@cumin1003: START - Cookbook sre.dns.netbox * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.move-vlan for host wikikube-worker1261 * 22:12 jasmine@cumin1003: START - Cookbook sre.hosts.reimage for host wikikube-worker1261.eqiad.wmnet with OS trixie * 22:11 jasmine@cumin1003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1261.eqiad.wmnet * 22:11 jasmine@cumin1003: START - Cookbook sre.k8s.renumber-node Renumbering for host wikikube-worker1261.eqiad.wmnet * 21:40 sbassett: Deployed security mitigations for [[phab:T432713|T432713]], [[phab:T435026|T435026]] * 20:47 jforrester@deploy1003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply * 20:47 jforrester@deploy1003: helmfile [staging] START helmfile.d/services/wikifunctions: apply * 20:37 cdobbins@cumin1003: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp5022.eqsin.wmnet with OS trixie * 20:31 krinkle@deploy1003: Finished scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] (duration: 17m 00s) * 20:29 lerickson@deploy1003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply * 20:25 krinkle@deploy1003: krinkle: Continuing with deployment * 20:22 lerickson@deploy1003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply * 20:18 krinkle@deploy1003: krinkle: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. * 20:14 krinkle@deploy1003: Started scap sync-world: Backport for [[gerrit:1330486{{!}}Fix incorrect number of children in m(under{{!}}over) (T435705)]] * 19:41 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance replace-disks -n ganeti1057.eqiad.wmnet krb1004.eqiad.wmnet [[phab:T435873|T435873]] * 19:39 dancy@deploy1003: Finished scap sync-world: testing (duration: 03m 00s) * 19:36 dancy@deploy1003: Started scap sync-world: testing * 19:35 dancy@deploy1003: sync-world failed: <RuntimeError> dictionary changed size during iteration (scap version: 4.286.0) (duration: 01m 12s) * 19:34 dancy@deploy1003: Started scap sync-world: testing * 19:34 inflatador: [bking@ganeti1046] ~$ sudo gnt-instance migrate krb1004 ganeti1035 -> ganeti1029 [[phab:T435873|T435873]] * 19:28 dancy@deploy1003: Finished scap sync-world: testing (duration: 05m 45s) * 19:23 cdobbins@cumin1003: END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.move-vlan for host cp5022 * 19:23 cdobbins@cumin1003: START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie * 19:22 dancy@deploy1003: Started scap sync-world: testing * 19:22 dancy@deploy1003: Installation of scap version "4.286.0" completed for 3 hosts * 19:20 dancy@deploy1003: Installing scap version "4.286.0" for 3 host(s) * 18:37 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/zotero: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply * 18:36 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply * 18:35 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply * 18:34 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply * 18:34 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/termbox: apply * 18:33 blake@deploy1003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply * 18:25 blake@deploy1003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply == Other archives == See [[Server Admin Log/Archives]]. <noinclude> [[Category:SAL]] [[Category:Operations]] </noinclude> lw4h53md0xyksovf2504rw4w37nwmso 0 17290 2452790 2452211 2026-08-31T04:18:07Z Stashbot 7414 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1332530 2452790 wikitext text/x-wiki === 2026-08-31 === * 04:18 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1332530 === 2026-08-27 === * 15:06 Krinkle: Disable duplicate publishing noise from extension-IPReputation, EIPR, [[phab:T143162|T143162]] === 2026-08-26 === * 18:07 andrewbogott: resolving rebase conflicts in /srv/git/labs/private * 17:24 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1329617 * 14:45 hashar: Updating Castor (0.4.3..0.4.5) on all Jenkins jobs {{!}} https://gerrit.wikimedia.org/r/1329577 * 07:11 hashar: integration: updated Quibble jobs to replace deprecated `--commands` option by 1..n `--command` option(s) {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1293711 {{!}} [[phab:T321617|T321617]] === 2026-08-25 === * 21:32 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 * 21:23 brett: revert deployment-prep acme-chief switch: acme-chief active to deployment-acme-chief05 and passive to deployment-acme-chief06 * 21:03 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 * 20:31 Southparkfan: [[phab:T401839|T401839]] - provisioned deployment-docker-wikifeeds01 (trixie) using Tofu (cloudvps-repos/deployment-prep/tofu-provisioning) * 15:06 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 ([[phab:T435368|T435368]]) * 07:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance deployment-ircd03 * 07:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance deployment-ircd03 === 2026-08-24 === * 08:29 hashar: integration: add firewall rule to ssh from contint1003/contint2003 IPv6 (IPv4 was already allowed) # [[phab:T435756|T435756]] * 08:26 hashar: integration: removing firewall rule for ssh from contint1002/contint2002 IPv4 # [[phab:T418521|T418521]] === 2026-08-21 === * 13:07 hashar: integration: triggered Doxygen doc for PersonalDashboard extension using: `zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/PersonalDashboard --change {{Gerrit|1328044}},1` # [[phab:T435392|T435392]] === 2026-08-20 === * 23:26 bd808: deployment-mediawiki14: `systemctl stop php8.3-fpm; sleep 5m; systemctl start php8.3-fpm` -- maybe the bot storm will break if we give fast 500 responses for 5 minutes. * 14:55 James_F: jforrester@integration-castor06:$ sudo rm -rf /srv/castor/castor-mw-ext-and-skins/master/wikilambda-catalyst-end-to-end # Clear stale Catalyst castor npm downloads. * 13:35 James_F: Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst * 08:54 hashar: deployment-prep: hard reboot deployment-cache-test-08 # [[phab:T435421|T435421]] === 2026-08-19 === * 14:48 hashar: integration: updated castor save job to have rsync emit statistics {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1321049 {{!}} [[phab:T432685|T432685]] * 12:30 James_F: Updating development images on contint primary for “fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image” === 2026-08-18 === * 17:32 dancy: Rebooting deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud for good measure * 17:31 dancy: rm /var/log/apache2/*.gz on deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud to free up ~6GB. * 09:24 hashar: zuul: restarted zuul-web === 2026-08-17 === * 21:09 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/115 * 21:01 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/118 ([[phab:T413817|T413817]], [[phab:T407430|T407430]]) * 18:08 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/113 * 16:30 James_F: Zuul: [mediawiki/extensions/WP25EasterEggs] Archive repository, for [[phab:T418134|T418134]] === 2026-08-14 === * 18:59 James_F: Zuul: Enforce CI for mediawiki-php-<nowiki>{</nowiki>excimer,luasandbox,wikidiff2<nowiki>}</nowiki> for [[phab:T425943|T425943]] * 18:20 James_F: Docker: [php85] Migrate to Wikimedia-provide binary, cascaded, for [[phab:T433254|T433254]] * 14:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1325900 === 2026-08-13 === * 19:55 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 ([[phab:T401115|T401115]]) (redux for typo fix) * 19:27 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 ([[phab:T401115|T401115]]) * 03:05 TimStarling: created Produnto tables on beta [[phab:T421436|T421436]] === 2026-08-12 === * 12:31 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/mwext-phpunit-coverage*/npm` # [[phab:T427922|T427922]] * 11:59 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/*codehealth*/npm` # [[phab:T427822|T427822]] === 2026-08-07 === * 16:13 hashar: integration: deleted integration-agent-[[phab:T422258|T422258]] agent # [[phab:T422258|T422258]] * 14:31 hashar: integration: updating Quibble jobs to Quibble 1.19.0 # [[phab:T432934|T432934]] [[phab:T432943|T432943]] [[phab:T427922|T427922]] * 07:25 hashar: Tag Quibble 1.19.0 @ {{Gerrit|a8a84ed1c0adb34688ce22066ebde8a2d584a7d4}} # [[phab:T432934|T432934]] [[phab:T432943|T432943]] [[phab:T427922|T427922]] === 2026-08-06 === * 18:02 dancy: Restarted gitlab-webhooks ([[phab:T430410|T430410]]) (revert) * 17:55 dancy: Restarted gitlab-webhooks ([[phab:T430410|T430410]]) === 2026-08-05 === * 20:31 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1184176 * 18:39 James_F: Docker: [php85] Upgrade PHP to 8.5.9 * 15:58 James_F: Zuul: [mediawiki/extensions/WikiLambda] Disable Catalyst, corrupted npm cache === 2026-08-04 === * 19:06 dancy: Updating buildkitd to v0.32.2 on gitlab-cloud-runners (staging and production) ([[phab:T433985|T433985]]) * 16:48 James_F: Docker: [php83] Upgrade PHP to 8.3.33 === 2026-08-03 === * 20:45 dancy: Updating buildkitd to v0.32.1 on gitlab-cloud-runners (staging and production) ([[phab:T433879|T433879]]) * 16:40 hashar: gerrit: added Vaughn Walters to integration group until he get added to the ciadmin LDAP group {{!}} [[phab:T433615|T433615]] * 16:37 James_F: Zuul: Drop REL1_44 testing, EOL, for [[phab:T428911|T428911]] === 2026-07-31 === * 18:14 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/114 * 12:23 hashar: integration: sudo cumin --force -p 0 'name:docker' 'rm -fR /srv/jenkins/workspace/*pipeline*' === 2026-07-30 === * 17:28 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # [[phab:T424419|T424419]] === 2026-07-29 === * 21:33 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 * 18:33 dancy: Buildkit v0.32.0 deployed to gitlab-cloud-runners staging and production ([[phab:T433520|T433520]]) === 2026-07-24 === * 17:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1316011 === 2026-07-23 === * 15:06 dancy: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/1314124 ([[phab:T295351|T295351]]) === 2026-07-22 === * 21:32 dancy: Moved /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/npm to /srv/castor-debug-[[phab:T20260722|T20260722]]-npm-torn-cacache/ on integration-castor06 * 18:41 dancy: Zuul dependencies upgraded and Zuul restarted. * 18:25 dancy: Zuul is currently broken due to the Gerrit SSH key update. I'm investigating * 17:24 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1314014/1 ([[phab:T432886|T432886]]) * 16:58 dancy: Restarting Gerrit ([[phab:T240266|T240266]]) (again) * 15:59 dancy: Restarting Gerrit ([[phab:T240266|T240266]]) * 09:54 James_F: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure for {{Gerrit|1313321}}, per Lucas_WMDE. === 2026-07-21 === * 15:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1313220 * 07:08 hashar: integration: cleaned up Jenkins workspace on integration-agent-docker-1084 === 2026-07-20 === * 19:36 dancy: Restarting jenkins on contint1003 to clear out hours-long stuck jobs === 2026-07-16 === * 23:07 mutante: gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed [[phab:T398401|T398401]] but caused [[phab:T432413|T432413]] - replication is working again * 18:51 dancy: Updated buildkit to v0.31.2 gitlab-cloud-runners (staging and production) ([[phab:T432360|T432360]]) * 18:37 dancy: Restarted Jenkins to unstick jobs. * 18:33 dancy: Jenkins placed in shutdown mode in preparation for a restart * 17:07 bd808: Hard reboot of deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud via Horizon; console shows OOM ([[phab:T432374|T432374]]) * 16:40 brennen: contint1002: chown and chmod on /srv/zuul/git/mediawiki/extensions/CampaignEvents/.git per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions * 16:34 dancy: Restarted docker on contint2003 ([[phab:T432326|T432326]]) * 16:33 dancy: Restarted docker on contint1003 ([[phab:T432326|T432326]]) * 16:32 dancy: Restarted docker on contint1003 * 12:31 Krinkle: krinkle@contint1003:~$ sudo /usr/sbin/service jenkins restart * 12:10 Krinkle: krinkle@contint1002: zuul restart === 2026-07-15 === * 18:58 James_F: dev-images: Re-build PHP images for latest point releases, for [[phab:T431099|T431099]] * 09:13 James_F: Docker: [php83] Update PHP to 8.3.32 for [[phab:T431099|T431099]] === 2026-07-14 === * 11:53 James_F: Docker: [commit-message-validator] Update to v3.0.0, for [[phab:T431799|T431799]] * 08:37 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud === 2026-07-13 === * 15:03 dancy: Updating gitlab-cloud-runners to v19.0.2 === 2026-07-10 === * 11:56 James_F: Zuul: Disable all browser tests on release branches except Wikibase's, for [[phab:T430415|T430415]] * 09:16 James_F: Docker: [commit-message-validator] Update to 2.3.0 === 2026-07-09 === * 11:55 hashar: retriggering postmerge change for [[phab:T431582|T431582]]: zuul enqueue --trigger gerrit --pipeline postmerge --project machinelearning/liftwing/inference-services --change {{Gerrit|1308631}},3 === 2026-07-08 === * 20:58 brennen: patchdemo: deployed https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/367 ([[phab:T427964|T427964]]) * 19:21 mutante: gerrit - replacing registerEmailPrivateKey in Gerrit config - this invalidates pending/outstanding email validation links for gerrit users - but does not affect active accounts or already verified email addresses * 14:36 hashar: contint1003, contint2003: manually installed `docker-buildx` Debian package to validate https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308659 # [[phab:T431582|T431582]] * 13:04 hashar: deployment-prep: git repack on /srv/mediawiki-staging/php-master * 12:27 hashar: deployment-prep: on deployment server: clearing old branches for mediawiki/extensions and mediawiki/skins # [[phab:T428864|T428864]] === 2026-07-07 === * 19:41 mutante: contint1003/2003 - add jenkins-agent user to docker group; restart jenkins * 17:17 hashar: gerrit: deleted /srv/gerrit/java_pid3571660.hprof === 2026-07-06 === * 20:06 dancy: Updated buildkitd to v0.31.1 in gitlab-cloud-runners ([[phab:T429988|T429988]]) * 08:11 James_F: Zuul: Add WikimediaAntiAbuse extension, for [[phab:T431023|T431023]] === 2026-07-03 === * 15:25 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too * 15:09 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep * 11:22 James_F: Zuul: Make the in-mediawiki-tarball template real * 10:06 James_F: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches * 09:53 James_F: Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18, for [[phab:T423987|T423987]] and [[phab:T425807|T425807]] === 2026-07-02 === * 13:27 James_F: Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade, for [[phab:T428570|T428570]] * 10:47 hashar: zuul1002: running Puppet agent to drop `wikimediacloud.org` from `no_proxy` {{!}} [[phab:T430479|T430479]] === 2026-07-01 === * 13:39 hashar: integration: added timestamping to operations-puppet-catalog-compiler and operations-puppet-catalog-compiler-puppet7-test jobs * 02:45 hashar: gerrit: on gerrit2003 deleted /srv/gerrit/java_pid3520115.hprof (the JVM apparently died at some point, I assume due to heavy crawling) === 2026-06-30 === * 15:13 dancy: Rebooting deployment-mwlog02.deployment-prep to clear stuck udp2log processes === 2026-06-29 === * 09:32 hashar: gerrit: deleted repository phabricator/extensions/BurnDownCharts , created in July 2014, had no commit/changes === 2026-06-28 === * 15:30 hashar: Updated integration/zuul-jobs from upstream (c75fe6ef19c..fc4af6d4471), notably to remove `requestsexceptions` in `upload-logs-swift` role # [[phab:T430458|T430458]] === 2026-06-26 === * 13:59 Krinkle: [[phab:T429658|T429658]] krinkle@doc1004:/srv/doc/cover-extensions$ sudo -u doc-uploader rm -rf ShortUrl/ * 13:59 Krinkle: [[phab:T429658|T429658]] krinkle@doc2003:/srv/doc/cover-extensions$ sudo -u doc-uploader rm -rf ShortUrl/ === 2026-06-25 === * 20:57 dancy: Restarting Jenkins to unstick builds * 20:49 dancy: Investigating castor-save-workspace-cache clog * 16:50 inflatador: add 60GB cinder vol to deployment-cirrussearch15 [[phab:T425585|T425585]] * 14:25 inflatador: delete unused servers deployment-cirrussearch1[2-4] [[phab:T425585|T425585]] * 10:07 James_F: Docker: Bump Node 24 / Node 26 to new releases === 2026-06-24 === * 17:26 dancy: Set `profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py` in hiera config for deployment-puppetserver prefix ([[phab:T429413|T429413]]) * 15:12 dancy: sudo systemctl restart php8.3-fpm on deployment-jobrunner05 (Attempting to resolve logspam) === 2026-06-23 === * 14:31 brennen: deploying patchdemo for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/361 === 2026-06-22 === * 23:19 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/mediawiki-core/master/mediawiki-node24/ #[[phab:T429824|T429824]] * 23:07 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ #[[phab:T429824|T429824]] (again) * 22:35 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ #[[phab:T429824|T429824]] * 19:30 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-with-gated-extensions-vendor-mysql-php83 #[[phab:T429824|T429824]] === 2026-06-19 === * 20:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1304632 * 09:39 Lucas_WMDE: deployment-deploy04: used createAndPromote to restore User:Lucas Werkmeister (WMDE) to bureaucrat (dewiki, enwiki, metawiki, wikidatawiki) and wikidata-staff (wikidatawiki) after groups were very unhelpfully removed, presumably due to 2FA enforcement, without apparent warning or announcement of any kind * 07:07 dcausse: restarted php-fpm on deployment-jobrunner05 (inconsistent php state: MediaWiki\Extension\PageAssessments\HookHandler\ParserHooks::__construct(): Argument #3 ($config) must be of type MediaWiki\Config\Config, MediaWiki\Page\WikiPageFactory given) * 06:13 hashar: Upgrading Quibble jobs to 1.18.2 (retry git connections on reset) https://gerrit.wikimedia.org/r/c/integration/config/+/1304181 # [[phab:T420865|T420865]] === 2026-06-18 === * 19:52 dancy: Building quibble 1.18.2 images on contint primary ([[phab:T420865|T420865]]) * 18:50 dancy: Tag Quibble 1.18.2 @ {{Gerrit|152b497d317eaafc3cec334d5ce7e549697a2980}} # [[phab:T420865|T420865]] * 15:15 dancy: Deleting deployment-db11 and deployment-db14 ([[phab:T428910|T428910]]) * 07:23 dcausse: reindexing all wikis to opensearch2 ([[phab:T425585|T425585]], [[phab:T427196|T427196]]) === 2026-06-17 === * 23:16 dduvall: restarted zuul to clear up 5 hrs of stuck queues * 23:03 mutante: re-enabled puppet on contint1003 - triple checked puppet does NOT start jenkins anymore. BOTH masked AND stopped while running untouched on contint1002. after: gerrit:1303578 {{!}} ([[phab:T418521|T418521]]) ([[phab:T428791|T428791]]) * 18:51 dduvall: performed stop/start of jenkins service on contint1002 following failed safeRestart * 18:45 dduvall: restarting jenkins due to stuck zuul queues * 14:56 inflatador: mwscript /srv/mediawiki-staging/php-master/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki=enwikibooks === 2026-06-16 === * 16:00 dancy: sudo keyholder arm on deployment-deploy04.deployment-prep * 15:48 dancy: Resizing deployment-deploy04.deployment-prep from g4.cores4.ram8.disk20 to g4.cores8.ram16.disk20 ([[phab:T429364|T429364]]) * 15:31 dancy: Turning off deployment-db11 and deployment-db14 === 2026-06-15 === * 23:40 dancy: systemctl restart php8.3-fpm on deployment-jobrunner05 to reload beta db configuration ([[phab:T428930|T428930]]) * 21:00 dancy: deployment-db15 promoted to master ([[phab:T428930|T428930]]) * 20:53 dancy: deployment-db11.deployment-prep going read-only * 20:48 dancy: deployment-db11.deployment-prep will be going read-only soon while db15 is being promoted to primary. * 20:26 dancy: Added deployment-db16 ([[phab:T429245|T429245]]) * 18:26 dancy: Rebooting deployment-jobrunner05 === 2026-06-12 === * 21:45 dancy: Unstuck wmf-beta-update-all service on deployment-deploy04.deployment-prep (sudo systemctl stop wmf-beta-update-all) * 18:00 thcipriani: unmasking jenkins on contint1002 and restarting * 17:49 thcipriani: attempting to cancel castor-save-workspace-cache {{Gerrit|6710545}} * 15:19 James_F: Docker: [php83] Re-platform to Debian Bookworm, for [[phab:T383337|T383337]] * 15:07 dancy: deployment-db15 configured as a replica of deployment-db11 ([[phab:T428930|T428930]]) * 10:21 Krinkle: `krinkle@<nowiki>{</nowiki>doc1004,doc2003<nowiki>}</nowiki>:/srv/doc/mediawiki-core$ sudo -u doc-uploader rm -rf list/` - remove doc build for git-tag test. === 2026-06-11 === * 14:46 hashar: for minor in $(seq 21 42); do ./.tox/make-release/bin/python -u ./make-release/branch.py --delete --abandon --bundle '*' "REL1_$minor"; done; * 14:46 hashar: On all MediaWiki repos, converting old release branches up to REL1_42 included to tags. Last time I missed non wmf repo # [[phab:T380841|T380841]] {{!}} [[phab:T428864|T428864]] * 09:45 hashar: Converted mediawiki/core branches REL1_39, REL1_40, REL1_41, REL1_42 to tags # [[phab:T428864|T428864]] * 09:18 hashar: Converting REL1_42 branches to tags # [[phab:T428864|T428864]] * 09:18 hashar: Converting REL1_41 branches to tags # [[phab:T428864|T428864]] * 09:05 hashar: Converting REL1_40 branches to tags # [[phab:T428864|T428864]] * 08:56 hashar: Converting REL1_39 branches to tags # [[phab:T428864|T428864]] * 08:40 hashar: gerrit: deleted mediawiki/core branch "development" that pointed to {{Gerrit|7f622781cc31053b121f6f4ddbff506cba10d38e}} (which is contained by master). Had probably been created by a direct push. === 2026-06-10 === * 16:06 James_F: Docker: Provide quibble-bookworm, for [[phab:T362705|T362705]] === 2026-06-04 === * 23:39 jeena: Updating development images on contint primary for [[phab:T424691|T424691]] * 12:30 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc # fix failure seen in mwext-node24-rundoc 4812 * 08:35 hashar: Built Docker images `docker-registry.wikimedia.org/releng/java21:0.1` and `docker-registry.wikimedia.org/releng/maven-java21:0.1` # [[phab:T412978|T412978]] === 2026-06-03 === * 14:19 jnuche: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/107 * 12:54 James_F: Zuul: Add Rae 5e as a trusted user * 08:26 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1296559 "inference-services: Add LLM generated editing suggestions CI/CD pipelines." # [[phab:T427794|T427794]] === 2026-06-02 === * 23:23 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1296695 * 23:10 thcipriani: tag quibble 1.18.1 @ {{Gerrit|4b7959553c095d811426f394165c69ecc13a44eb}} * 18:13 brennen: devtools phab/phorge: deployed work/2026-06-01-merge-phorge to https://phabricator.wmcloud.org/ for testing ([[phab:T410849|T410849]]) * 16:05 hashar: integration: upgraded pypy from 7.3.11 (py3.9) to 7.3.20 (py3.11) # [[phab:T423607|T423607]] * 15:49 jnuche: Updating buildkitd to v0.30.0 in gitlab-cloud-runners ([[phab:T426212|T426212]]) * 15:24 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/1295865 # [[phab:T423607|T423607]] * 14:57 jnuche: Jenkins/Zuul is back * 14:38 jnuche: restarting Jenkins * 14:28 jnuche: bring back castor node, that didn't help * 14:23 jnuche: trying to reconnect castor node, see if that helps somehow * 14:12 jnuche: option to "Enable Gearman" times out. Can't re-enable from UI. Gearman plugin logs are empty. Neat * 14:05 jnuche: trying to reconnect Gearman === 2026-06-01 === * 21:34 jeena: Updating development images on contint primary for [[phab:T424663|T424663]] * 10:58 hashar: gerrit: flushed `ldap_usernames` cache in case a missing account ended up being cached there # [[phab:T427792|T427792]] === 2026-05-28 === * 16:48 hashar: castor: nuked SonarQube cache: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-codehealth-master-non-voting/sonar/ # [[phab:T427471|T427471]] * 07:05 hashar: integration: delete deployment-deploy04 agent from Jenkins controller. Batch job has been migrated to a systemd driven script # [[phab:T256168|T256168]] * 06:29 hashar: integration: delete integration-castor05 agent from Jenkins, replaced by integration-castor06 # [[phab:T421114|T421114]] === 2026-05-27 === * 21:07 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1294425 * 09:49 hashar: Updated helm-lint Jenkins job to use releng/helm-linter:0.8.0 image # [[phab:T424824|T424824]] * 08:25 codders: integration-castor06: sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ === 2026-05-26 === * 15:26 James_F: Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops * 14:46 hashar: Updating Quibble Jenkins jobs to stop Supervisord from spawning Memcached # [[phab:T397810|T397810]] * 14:07 hashar: Updated integration-quibble-* jobs in order to validate running tests with Supervisord not managing memcached # [[phab:T397810|T397810]] * 13:15 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 981, 985, 987 * 09:05 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1292620 (Introduce Phan composer job - [[phab:T231966|T231966]]) === 2026-05-21 === * 16:33 hashar: Reloaded Zuul to enable Node24 CI job for `labs/tools/wdaudiolex-fe` # [[phab:T426366|T426366]] === 2026-05-19 === * 18:07 James_F: Zuul: [mediawiki/libs/ZestJQ] Add basic PHP and Node CI === 2026-05-18 === * 21:12 James_F: Zuul: [operations/software/gerrit] Add Node26 as experimental * 21:10 mutante: gerrit-replica.wikimedia.org, gerrit-spare.wikimedia.org - rebooting backends * 20:57 James_F: Zuul: [integration/docroot] Test in PHP 8.3+, dropping 8.2 * 20:56 James_F: Zuul: [analytics/wmde/scripts] Test in PHP 8.3+, dropping 8. * 20:18 James_F: Zuul: [wikimedia/fundraising/dash] Replace Node 20 testing with Node 24 * 20:18 James_F: Zuul: Migrate various labs things to Node 24 * 20:02 James_F: Docker: [ajv, sonar-scanner] Migrate to Node 24 * 19:58 James_F: Zuul: Migrate various production/CI things to Node 24 * 18:17 mutante: releases.wikimedia.org - rebooting backends * 18:14 mutante: rebooting production gitlab-runners * 18:12 dancy: gitlab-cloud-runners have been revived. * 18:11 James_F: Zuul: [design/codex] Switch CI to Node 24 * 15:52 dancy: gitlab-cloud-runners are in a broken state. I'm investigating * 14:27 hashar: Upgrading Quibble jobs to 1.18.0 * 09:29 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 22260 === 2026-05-15 === * 18:36 dancy: Upgraded gitlab-cloud-runners (prod) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 18:24 dancy: Upgrading gitlab-cloud-runners (prod) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 18:11 dancy: Upgraded gitlab-cloud-runners (staging) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 17:59 dancy: Upgrading gitlab-cloud-runners (staging) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 13:02 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1287828 [[phab:T426392|T426392]] === 2026-05-13 === * 12:42 James_F: Zuul: [mediawiki/extensions/Springboard] Add AdminLinks Phan dependency * 12:42 James_F: Zuul: [mediawiki/extensions/ChatBot] Add dependencies on VisualEditor and BlueSpiceFoundation * 12:42 James_F: Zuul: [mediawiki/extensions/ChatIntegration] Add dependency on VisualEditor * 12:37 James_F: Zuul: [mediawiki/extensions/WikiLambda] Drop AF and SB deps down to phan-only, for [[phab:T423180|T423180]] === 2026-05-12 === * 20:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/104 ([[phab:T424774|T424774]]) * 18:08 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add AF and SB deps for [[phab:T423180|T423180]] * 14:18 atsukoito: PrivateSettings: empty $wgOpensearchCredentials for opensearch-on-k8s synced to deploy04 by Reedy * 13:04 atsukoito: PrivateSettings: credentials for opensearch-on-k8s ttmserver-test * 11:50 James_F: Zuul: [machinelearning/liftwing/inference-services] Add qwen36 llm model CI/CD pipelines, for [[phab:T425680|T425680]] * 11:46 James_F: Zuul: Add experimental php-pie-build* jobs to other PHP extensions, for [[phab:T425943|T425943]] * 11:37 James_F: Zuul: [mediawiki/php/wikidiff2] Add experimental php-pie-build* jobs, for [[phab:T425943|T425943]] * 10:05 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 # fix failure seen in quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 7817 * 08:44 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51633 === 2026-05-11 === * 18:28 James_F: Docker: Add changes to php-compile images for PIE, for [[phab:T425943|T425943]] * 16:06 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51439 and 51452 === 2026-05-09 === * 20:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1285498 === 2026-05-07 === * 22:53 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/105 === 2026-05-06 === * 18:13 bd808: Unblock 88.165.192.0/19 * 18:03 bd808: Unblock 94.208.0.0/14 * 17:56 bd808: Unblock 84.226.0.0/16 * 17:41 bd808: Unblock 94.34.0.0/16 * 17:35 bd808: Unblock 109.134.0.0/16 === 2026-05-05 === * 21:20 James_F: Zuul: Provide Node 26 experimental jobs everywhere needed * 21:04 James_F: Docker: Provide initial Node 26 images * 19:01 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto dependency, for [[phab:T396135|T396135]] * 14:58 dancy: rm /var/log/<nowiki>{</nowiki>user.log.1,syslog.1,messages.1<nowiki>}</nowiki> on deployment-eventgate-4.deployment- prep ([[phab:T425429|T425429]]) === 2026-05-04 === * 15:19 dancy: Upgrading gitlab cloud runners (prod) from 1.35.1-do.3 to 1.35.1-do.5 * 14:51 dancy: Upgrading gitlab cloud runners (staging) from 1.35.1-do.3 to 1.35.1-do.5 * 10:40 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template === 2026-05-02 === * 20:49 James_F: Zuul: [mediawiki/core] Enforce PHP 8.4 & 8.5 on release branches, all pass * 19:27 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for MW release branches * 19:19 James_F: Zuul: [mediawiki/extensions/BlogPage] Add dependencies * 16:48 James_F: Hard-restarting Zuul to clear the huge number of i18n updates being re-submitted. * 15:48 James_F: Zuul: [wikimedia-cz/*] Test in PHP 8.3+, dropping 8.2 * 14:02 TheresNoTime: Add bvibber to deployment-prep project * 09:08 James_F: Docker: [quibble-*] Add php-luasandbox so we can test both modes in Scribunto === 2026-05-01 === * 15:42 James_F: Zuul: [wikimedia/lucene-explain-parser] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: Zuul: [wikimedia/textcat] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: Zuul: [mediawiki/tools/ParseWiki] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: zuul: Add ToprakM to CI allowlist * 15:19 James_F: Zuul: [translatewiki] Test in PHP 8.3+, dropping 8.2 * 15:10 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add TestKitchen as a dependency, for [[phab:T425076|T425076]] * 12:40 James_F: Zuul: [mediawiki/tools/code-utils] Test in PHP 8.3+, dropping 8.2 * 08:02 James_F: Zuul: Update xtex's e-mail in the allowlist * 07:37 James_F: Zuul: Switch release branches' selenium jobs to PHP 8.3 * 07:33 James_F: Zuul: Test Wikimedia production libraries in PHP 8.3+, dropping 8.2 === 2026-04-30 === * 21:36 brennen: gitlab-webhooks: building & restarting to deploy https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 * 20:26 James_F: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting * 20:16 James_F: jforrester@doc1004:~$ # sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn/ # [[phab:T415832|T415832]] * 20:14 James_F: Zuul: [mediawiki/extensions/WebAuthn] Archive, for [[phab:T415832|T415832]] / [[phab:T303495|T303495]] * 17:16 brennen: wikibugs: most maintainers at hackathon, so go release-engineering added as a maintainer while looking to debug error at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/810904 * 15:19 mutante: upgrading zuul to 14.2.0-1 on "new zuul" machines ([[phab:T424879|T424879]]) === 2026-04-29 === * 15:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add ConfirmEdit dependency, for [[phab:T424597|T424597]] * 15:36 James_F: Zuul: Drop experimental node22 jobs, never used in practice * 15:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1279392, https://gerrit.wikimedia.org/r/1279397 === 2026-04-28 === * 18:11 bd808: Unblock 86.0.0.0/16 * 17:41 bd808: Unblock 79.192.0.0/10 * 17:07 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing * 17:07 James_F: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing * 17:07 James_F: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing * 16:32 James_F: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing * 13:34 James_F: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing * 13:34 James_F: Zuul: [oojs/ui] Drop PHP 8.2 testing * 13:14 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI * 10:40 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717 * 00:03 bd808: Increase parallelism for wmf-beta-update-databases.py ([[phab:T256168|T256168]]) === 2026-04-27 === * 22:11 bd808: Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ ([[phab:T256168|T256168]]) * 21:57 bd808: Add web security group to deployment-deploy04 ([[phab:T256168|T256168]]) * 20:45 James_F: Zuul: Restrict mw*-codehealth-patch jobs to master only, for [[phab:T424573|T424573]] * 17:16 James_F: Docker: [mediawiki-phan-taint-check-demo] Re-platform to Trixie and so PHP 8.4 * 15:53 James_F: Zuul: [mediawiki/extensions/ReportIncident] Add TestKitchen phan dependency, for [[phab:T424220|T424220]] * 14:32 James_F: Zuul: Drop PHP 8.2 enforcement from MediaWiki things for master and REL1_46 for [[phab:T358667|T358667]] * 12:38 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 383 * 09:18 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # [[phab:T424419|T424419]] === 2026-04-26 === * 20:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276777 === 2026-04-24 === * 22:48 dduvall: merged zuul3 branch of integration/config into master and pushed (in preparation for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198) * 12:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276428 === 2026-04-23 === * 23:57 bd808: Set `profile::beta::autoupdater::run_updater: true` for deployment-deploy04 via Horizon ([[phab:T256168|T256168]]) * 22:58 bd808: bd808@deployment-deploy04 `sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-all` * 22:36 bd808: bd808@deployment-deploy04 `sudo -u mwdeploy /usr/local/bin/wmf-beta-update-all` * 22:16 bd808: Disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad so that replacement script can be tested ([[phab:T256168|T256168]]) * 22:12 bd808: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad so that replacement script can be tested ([[phab:T256168|T256168]]) * 22:02 bd808: Cherry-picked {{gerrit|1276813}} to deployment-puppetserver-1 ([[phab:T256168|T256168]]) * 20:11 James_F: Zuul: [wikibase/*] Replace CI testing in Node 20 with Node 24 * 20:11 James_F: Zuul: [wikidata/query/*] Replace CI testing in Node 20 with Node 24 * 20:11 James_F: Zuul: [analytics/*] Replace CI testing in Node 20 with Node 24 * 20:10 James_F: Zuul: [mediawiki/tools/*] Replace CI testing in Node 20 with Node 24 * 20:06 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]]) * 19:55 James_F: Zuul: [jquery-client] Replace CI testing in Node 20 with Node 24 * 19:51 James_F: Zuul: [wikipeg] Drop testing in Node 20 and Node 22 * 19:47 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]]) * 19:37 James_F: Zuul: [oojs/ui] Drop CI testing in Node 20 and Node 22 * 19:37 James_F: Zuul: [oojs/js] Drop CI testing in Node 20 and Node 22 * 19:37 James_F: Zuul: [unicodejs] Replace CI testing in Node 20 with Node 24 * 19:36 James_F: Zuul: [wikimedia/portals] Drop CI testing in Node 20 and Node 22 * 18:57 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]]) * 18:39 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]]) * 18:18 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]]) * 17:58 James_F: Zuul: [mediawiki/extensions/OAuth] Add dependency on CentralAuth, for [[phab:T415281|T415281]] * 17:56 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.13-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]]) * 16:35 James_F: Zuul: Enforce PHP 8.5 CI for MW things in master (and REL1_46), for [[phab:T411814|T411814]] * 16:19 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.5 CI * 15:47 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add AntiSpoof dependency, for [[phab:T420548|T420548]] * 14:20 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 8385 and 8405 * 12:56 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add CentralNotice dependency, for [[phab:T422082|T422082]] === 2026-04-22 === * 00:07 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency, for [[phab:T424113|T424113]] === 2026-04-21 === * 23:26 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too, for [[phab:T394410|T394410]] * 23:17 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs, for [[phab:T422031|T422031]] * 20:47 inflatador: updating cirrussearch hosts to Trixie/OpenSearch 2 [[phab:T421763|T421763]] * 20:38 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep, for [[phab:T394410|T394410]] * 20:17 bd808: Running tofu for [[phab:T421244|T421244]] * 18:00 James_F: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency * 16:35 bd808: Unblock 79.116.0.0/16 * 13:34 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep, for [[phab:T415254|T415254]] * 13:27 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency, for [[phab:T420548|T420548]] === 2026-04-20 === * 23:56 bd808: Unblock 76.157.0.0/16 * 18:28 dancy: Upgrading gitlab cloud runners (staging) to 1.33.9-do.2 ([[phab:T423726|T423726]]) * 18:28 dancy: Upgrading gitlab cloud runners (staging) ([[phab:T423726|T423726]]) * 18:19 James_F: jjb: All 486 (!) jobs now updated for [[phab:T423622|T423622]] * 18:18 bd808: Unblock 113.128.0.0/15 * 15:03 James_F: Docker: Bump ci-bullseye/-bookworm/-trixie for mirrors.wm.org removal, [[phab:T423622|T423622]] === 2026-04-19 === * 19:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272752 === 2026-04-17 === * 21:07 thcipriani: marking integration-agent-1080 offline for experimentation * 19:30 thcipriani: reconfiguring castor-save-workspace-cache with https://gerrit.wikimedia.org/r/1273935 * 17:47 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]]) * 16:49 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]]) === 2026-04-16 === * 20:49 dduvall: creating integration/zuul-jobs repo to serve as a mirror of opendev.org/zuul/zuul-jobs ([[phab:T406384|T406384]]) * 13:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272711 [[phab:T423568|T423568]] * 11:07 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud === 2026-04-15 === * 20:05 James_F: Zuul: Configure REL1_46 CI, for [[phab:T423257|T423257]] * 17:44 bd808: Unblock 176.0.0.0/13 * 17:39 bd808: Unblock 46.128.0.0/16 * 17:32 bd808: Unblock 176.86.0.0/16 * 16:39 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/127d783b2176ac60b646a5fa4f1b1a872ca66340 * 15:33 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/100 * 01:02 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/99 === 2026-04-14 === * 20:42 James_F: Docker: [composer-scratch] Upgrade composer to 2.9.7 and cascade * 16:35 bd808: Unblock 88.112.0.0/14 * 00:48 bd808: Unblock 24.6.0.0/16 * 00:42 bd808: Unblock 152.231.48.0/20 === 2026-04-13 === * 22:00 James_F: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches * 20:28 James_F: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used * 14:42 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep, for [[phab:T421011|T421011]] === 2026-04-12 === * 18:21 James_F: jforrester@contint1002:~$ sudo /usr/sbin/service zuul restart && tail -f -n100 /var/log/zuul/zuul.log # [[phab:T423027|T423027]] === 2026-04-10 === * 23:22 James_F: jforrester@contint1002:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/ReadingLists --change {{Gerrit|1269498}},2 # [[phab:T422976|T422976]] * 23:20 James_F: Zuul: [mediawiki/extensions/ReadingLists] Publish JS coverage, for [[phab:T422976|T422976]] * 23:13 James_F: Zuul: Migrate a few straggler Node 20 MediaWiki things to Node 24 * 23:01 James_F: Zuul: Move all MediaWiki things from mediawiki-node20 to mediawiki-node24 * 21:59 James_F: Docker: Bump Node base images to March releases and cascade; Upgrade Quibble images from Node 20 to Node 24 * 10:24 hashar: Updating all Quibble jobs to 1.17.1 * 10:22 hashar: Updated PostgreSQL jobs to Quibble 1.17.1 # [[phab:T422110|T422110]] * 10:22 hashar: Updated apitesting job to Quibble 1.17.1 # [[phab:T422843|T422843]] [[phab:T418743|T418743]] * 09:51 hashar: Tag Quibble 1.17.1 @ {{Gerrit|0a1ab3b7c3dfee36c9bc2e9b049957d94e190e85}} === 2026-04-09 === * 15:13 hashar: Rolling back Quibble jobs to 1.16.0 (api-testing stage fails due to missing npm install step` * 14:58 hashar: Upgrading Quibble jobs to 1.17.0 * 14:23 hashar: Tagged Quibble 1.17.0 @ {{Gerrit|864381c6b63bdbcd8c74a3162c406fffcaaf8694}} * 07:48 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 "Zuul: use standalone jobs for GrowthExperiments Cypress tests" {{!}} [[phab:T417412|T417412]] === 2026-04-08 === * 22:19 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1269068 * 22:01 bd808: Unblock 95.216.12.170/32 ([[phab:T422751|T422751]]) * 19:26 brennen: gitlab-webhooks: building & deploying https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/37 - hitting some build tooling stuff, trying a fix per instructions in the error log * 17:54 bd808: Unblock 167.56.0.0/13 ([[phab:T422721|T422721]]) * 06:31 hashar: Deleted integration-agent-castor05 Bullseye instance, replaced by integration-agent-castor06 which is on Bookworm # [[phab:T421114|T421114]] * 06:24 hashar: Deleted integration-agent-qemu-1003 Bullseye image, replaced by integration-agent-qemu-1004 which is on Bookworm # [[phab:T422488|T422488]] === 2026-04-07 === * 22:25 dduvall: adding new pipelinelib labels to ci nodes ([[phab:T422234|T422234]]) * 20:05 hashar: Triggered a build of https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/ * 17:06 dduvall: added `Docker` label to `contint` jenkins nodes ([[phab:T422507|T422507]]) * 17:05 dduvall: restored missing `pipelinelib` labels on `integration-agent-docker-` CI hosts ([[phab:T422507|T422507]]) * 16:53 bd808: Unblock 73.0.0.0/8 ([[phab:T422498|T422498]]) * 12:36 hashar: jjb: use $CASTOR_HOST for Quibble success cache. https://gerrit.wikimedia.org/r/1268545 {{!}} This causes the Quibble jobs to use a new instance for the success cache, which is empty # [[phab:T383243|T383243]] [[phab:T421114|T421114]] * 12:17 hashar: Migrated Castor from integration-castor05 to integration-castor06. Updated CASTOR_HOST in Jenkins and moved the Cinder volume to the new instance # [[phab:T421114|T421114]] * 11:14 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1090, 1091, 1092 and 1093 # [[phab:T421114|T421114]] * 10:09 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1083 to 1089 # [[phab:T421114|T421114]] * 07:23 hashar: CI Jenkins: removed `blubber` label from all agents after having moved PipelineLib to use the `Docker` label {{!}} [[phab:T422234|T422234]] === 2026-04-06 === * 16:01 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1268239 === 2026-04-03 === * 20:17 bd808: Unblock 2.54.0.0/16 ([[phab:T422238|T422238]]) * 17:25 bd808: Unblock 31.18.0.0/16 ([[phab:T422245|T422245]]) * 17:18 bd808: Unblock 2.54.128.0/19 ([[phab:T422238|T422238]]) * 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1264649 "add Python 3.14 to pywikibot jobs and separate lint tests" {{!}} [[phab:T421723|T421723]] * 09:26 hashar: integration: nuked pywikibot/core pre-commit cache # [[phab:T422242|T422242]] * 09:15 hashar: Added Bookworm based Jenkins agents to the pool with label `Docker`. Hostnames are `integration-agent-docker-107*` # [[phab:T421114|T421114]] * 02:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1267398 === 2026-04-02 === * 16:50 thcipriani: restart jenkins * 15:15 bd808: Unblock 82.216.0.0/16 ([[phab:T421508|T421508]]) * 15:07 bd808: Unblock 95.90.0.0/15 ([[phab:T421485|T421485]]) * 11:19 James_F: Zuul: [oojs/ui] Drop ooui-ruby2.7-rake job, we're abandoning Ruby use there === 2026-04-01 === * 22:01 bd808: Unblock 109.144.0.0/12 ([[phab:T422019|T422019]]) * 20:16 bd808: Unblock 93.192.0.0/10 ([[phab:T421894|T421894]]) * 19:25 dancy: Updating buildkitd to v0.29.0 in gitlab-cloud-runners (prod) ([[phab:T415284|T415284]]) * 17:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/97 ([[phab:T420441|T420441]]) * 17:39 bd808: Unblock 94.134.0.0/15 ([[phab:T421866|T421866]]) * 16:31 dancy: Upgrade buildkit to 0.29.0 in staging gitlab-cloud-runners ([[phab:T415284|T415284]]) * 10:47 taavi: integration-castor05: free up a bit of disk space by deleting cache for AhoCorasick/ CLDRPluralRuleParser/ HtmlFormatter/ RelPath/ RunningStat/ IPSet/ === 2026-03-30 === * 22:01 bd808: Unblock 78.20.0.0/14 ([[phab:T421586|T421586]]) * 21:04 bd808: Unblock 95.88.0.0/15 ([[phab:T421774|T421774]]) * 20:49 bd808: Unblock 95.89.191.0/24 ([[phab:T421774|T421774]]) * 20:29 bd808: Unblock 73.162.0.0/16 ([[phab:T421549|T421549]]) * 13:10 hashar: gerrit: abandon mediawiki/core changes that are 2+years old and are attached to a task (`Bug: Txxxx`) * 11:37 hashar: Reloaded Zuul to to add 3 persons to the allow list * 10:43 James_F: Docker: Re-pushing to try to create quibble-coverage 1.16.0-s2 === 2026-03-27 === * 21:00 James_F: Docker: [quibble-bullseye] Drop Python 2 from images * 11:28 hashar: deployment-prep: removed block for `143.176.0.0/15` and blocked subblock `143.176.0.0/16` instead. This unblocks `143.177.0.0/16` # [[phab:T421420|T421420]] * 00:18 bd808: Unblock 95.90.238.0/23 ([[phab:T421447|T421447]]) === 2026-03-26 === * 21:25 bd808: Unblock 89.240.0.0/15 ([[phab:T421364|T421364]]) * 21:09 brennen: patchdemo: deploy to production for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312 === 2026-03-25 === * 20:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256318 [[phab:T421283|T421283]] * 15:46 dancy: Migrated gitlab-cloud-runners (prod) from nginx-ingress to traefik ([[phab:T420743|T420743]]) * 15:32 dancy: Migrated gitlab-cloud-runners (staging) from nginx-ingress to traefik ([[phab:T420743|T420743]]) * 10:01 hashar: Updating tox Jenkins jobs to add support for Python 3.14 {{!}} https://gerrit.wikimedia.org/r/1260632 {{!}} [[phab:T421209|T421209]] * 08:40 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20/ === 2026-03-24 === * 19:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255746 * 15:34 brennen: gitlab1004: manual test run of `configure-projects` with cleared issue allowlist ([[phab:T412882|T412882]]) * 15:26 bd808: Unblock 47.194.0.0/16 ([[phab:T421127|T421127]]) * 12:53 hashar: integration: deleted old Puppet 5 compiler agents from Jenkins ( pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud ) # [[phab:T367399|T367399]] * 07:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1259755 === 2026-03-23 === * 15:28 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 90272 === 2026-03-22 === * 14:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1258082 * 01:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256488 === 2026-03-21 === * 08:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256962 * 07:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256946 === 2026-03-20 === * 21:21 bd808: Unblock 103.159.218.0/24 ([[phab:T420530|T420530]]) * 14:59 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror, for [[phab:T399673|T399673]] === 2026-03-19 === * 16:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255777 * 16:01 Krinkle: Hoist l10n-bot rights from labs/tools parent to labs parent to reduce duplication in other labs/ repos * 15:50 Krinkle: Create labs/xtools repo (branch: main, parent: labs, owner: labs-xtools), ref [[phab:T402086|T402086]] === 2026-03-18 === * 21:11 dcausse: [[phab:T403775|T403775]]: reindexing all wikis to enable new sorting options * 21:08 dcausse: restarting opensearch on deployment-cirrussearch(12{{!}}13{{!}}14) instances to pickup new plugin versions * 14:56 James_F: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest * 14:52 James_F: Zuul: [GrowthExperiments] drop duplicate VisualEditor dep * 14:52 James_F: Zuul: [search/*] Add experimental Java 25 jobs === 2026-03-17 === * 22:50 James_F: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs * 21:27 James_F: Zuul: search: Update opensearch plugins for Java 11/17, for [[phab:T420407|T420407]] * 20:20 bd808: Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 ([[phab:T415021|T415021]]) * 16:43 James_F: Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps * 14:36 James_F: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep, for [[phab:T420015|T420015]] === 2026-03-13 === * 13:59 andrewbogott: deleting ptr record 117.0.16.172.in-addr.arpa. -- accidental duplicate for deployment-kafka-logging01.deployment-prep.eqiad1.wikimedia.cloud * 13:04 elukey: re-create kafka-logging-01 in deployment-prep on trixie and Kafka 3.7 (was running on buster) * 09:13 elukey: upgrade kafka-jumbo and kafka-main to Confluent 7.7 in deployment-prep (pre-requisite before being able to upgrade to Trixie) === 2026-03-12 === * 21:23 bd808: Hard reboot deployment-sessionstore06 ([[phab:T415021|T415021]]) * 01:14 James_F: Docker: [helm-linter] Bump for Envoy 1.35.9, for [[phab:T419637|T419637]] === 2026-03-11 === * 16:48 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/MetricsPlatform # [[phab:T417568|T417568]] * 16:47 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Archive, for [[phab:T416865|T416865]] * 11:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1250529 "inference-services: Split policy violation CI into separate model jobs." - [[phab:T418832|T418832]] === 2026-03-10 === * 17:39 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner production * 17:11 hashar: Updated MediaWiki coverage jobs so that they now keep "Generate a local configuration by running `composer phpunit:config`" message # [[phab:T419073|T419073]] * 16:41 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner staging * 08:21 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 === 2026-03-09 === * 21:53 bd808: Reboot deployment-shellbox01 on the off chance that is makes the new permissions error go away ([[phab:T419440|T419440]]) * 13:13 James_F: Zuul: [mediawiki/extensions/WikiShare] Mark as archived, for [[phab:T413589|T413589]] * 13:11 James_F: Zuul: [mediawiki/extensions/Memento] Mark as archived, for [[phab:T369991|T369991]] * 13:10 James_F: Zuul: [mediawiki/extensions/QuickGV] Mark as archived, for [[phab:T413348|T413348]] * 13:10 James_F: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived, for [[phab:T413588|T413588]] * 13:09 James_F: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived, for [[phab:T413587|T413587]] * 13:07 James_F: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived, for [[phab:T413586|T413586]] * 13:06 James_F: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived, for [[phab:T413585|T413585]] * 13:04 James_F: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived, for [[phab:T418008|T418008]] * 12:50 James_F: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency * 12:50 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen * 10:40 hashar: gerrit: mediawiki/vendor: converted `es6` and `es710` branches to tags # [[phab:T417804|T417804]] * 09:24 hashar: Updating Quibble jobs to 1.16.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1248880 {{!}} [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]] * 09:15 hashar: updating all CI Jenkins jobs using `./jjb-update` === 2026-03-06 === * 19:46 James_F: Zuul: [mediawiki/services/geoshapes] Mark as archived, for [[phab:T418372|T418372]] * 16:37 hashar: Building Docker images for Quibble 1.16.0 * 16:31 hashar: Tag Quibble 1.16.0 @ {{Gerrit|0b9db5fe3cabb2cec0b5d44e128bafa917b3b895}} # [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]] * 12:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248411 "jjb, Zuul: vary Wikibase Selenium for release branches" {{!}} [[phab:T418797|T418797]] * 12:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248409/ "jjb, Zuul: rename wikibase-selenium job for clarity" {{!}} [[phab:T418797|T418797]] === 2026-03-05 === * 14:41 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add TestKitchen as a dependency for [[phab:T418053|T418053]] * 08:01 hashar: Reloaded Zuul to rename wikibase-client / wikibase-repo jobs {{!}} https://gerrit.wikimedia.org/r/1238317 * 00:04 James_F: Docker: [quibble-coverage] Use local PHPUnit config, for [[phab:T345481|T345481]] === 2026-03-04 === * 21:16 James_F: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]] * 21:10 James_F: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]] * 19:48 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 ([[phab:T419004|T419004]]) * 18:50 James_F: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency", for [[phab:T419043|T419043]] * 16:23 James_F: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting * 15:37 James_F: Docker: [rake-ruby2.7] Add libffi-dev too, for [[phab:T418463|T418463]] * 13:59 James_F: Docker: [rake-ruby2.7] Add ruby-ffi for [[phab:T418463|T418463]] * 13:54 hashar: SIGKILL Zuul cause it can't gracefully stop most probably due to being locked attempting to report back to Gerrit # [[phab:T419009|T419009]] * 13:49 hashar: Stopping Zuul # [[phab:T419009|T419009]] * 13:41 hashar: Took a Zuul stack dump on contint1002.wikimedia.org using SIGUSR1 # [[phab:T419009|T419009]] === 2026-03-03 === * 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Drop MetricsPlatform phan dep * 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Drop MetricsPlatform phan dep === 2026-03-02 === * 22:13 James_F: Zuul: Enforce PHP 8.4 in MW extensions and skins for development branch, for [[phab:T386108|T386108]] * 14:05 James_F: Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency, for [[phab:T415451|T415451]] * 13:48 James_F: Zuul: […/WikimediaEvents] Drop LoginNotify dependency, now unused, for [[phab:T404334|T404334]] * 10:16 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/15.8.2/ # [[phab:T418718|T418718]] === 2026-02-28 === * 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/skins` # [[phab:T418675|T418675]] * 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/extensions` # [[phab:T418675|T418675]] === 2026-02-27 === * 15:53 dancy: Updating gitlab-cloud-runners (staging and prod) to gitlab-runner 18.9.0. === 2026-02-26 === * 20:16 James_F: Zuul: Provide a custom, high-priority pipeline just for puppet compiler [[phab:T414621|T414621]] * 19:32 James_F: Docker: Bump all the PHPs. * 13:40 hashar: Deployed Jenkins job https://integration.wikimedia.org/ci/job/wikibase-selenium/ # [[phab:T287582|T287582]] * 00:13 dduvall: forcing replacement of buildkitd helm release in gitlab-cloud-runner prod cluster due to dependency on removed k8s secret ([[phab:T416260|T416260]]) === 2026-02-25 === * 23:50 dduvall: deploying https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 to gitlab-cloud-runner production cluster ([[phab:T416260|T416260]]) * 14:07 James_F: Zuul: [mediawiki/extensions/CommunityRequests] Add TemplateData dependency, for [[phab:T401638|T401638]] * 00:08 jeena: no-op testing updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 === 2026-02-24 === * 15:55 brennen: devtools: test deploy phab/phorge to test instance ([[phab:T418256|T418256]]) === 2026-02-23 === * 23:07 jeena: Updated development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 * 22:43 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 * 22:12 bd808: Unblock 191.80.192.0/18 ([[phab:T418132|T418132]]) * 20:26 hashar: Deleted "replication-upstream" Grafana dashboard in favor of a copy/new "replication" one. https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream , replaced it by https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication * 16:29 James_F: Zuul: [3d2png] Add basic Node CI at version 20 === 2026-02-20 === * 21:47 bd808: Unblock 168.184.84.0/24 ([[phab:T418020|T418020]]) * 17:13 bd808: Unblock 122.187.64.0/18 ([[phab:T417964|T417964]]) * 14:35 James_F: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section === 2026-02-19 === * 18:34 bd808: Unblock 181.98.0.0/16 ([[phab:T417890|T417890]]) * 17:21 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add AbuseFilter as a dependency, for [[phab:T417799|T417799]] * 13:22 hashar: Reloaded Zuul to archive the Cergen repository {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1240688 {{!}} [[phab:T417887|T417887]] === 2026-02-18 === * 20:17 jeena: Updating development images on contint primary for [[phab:T415922|T415922]] * 19:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240360 * 18:40 bd808: Unblock 46.59.0.0/17 ([[phab:T417747|T417747]]) * 17:05 hashar: Regenerating Jenkins jobs with JJB based on https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/ * 17:04 hashar: Added EXT_DEPENDENCIES to Quibble Jenkins jobs parameters so we can manually trigger them from the Web UI using a different set of deps # https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/ * 16:30 hashar: Triggered https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ with empty Zuul parameters introduced by https://gerrit.wikimedia.org/r/1240333 {{!}} https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4893/console * 15:43 James_F: Zuul: [mediawiki/extensions/ReadingLists] Add EventBus dependency for [[phab:T417706|T417706]] * 12:15 hashar: zuul-1001.zuul3.eqiad1.wikimedia.cloud: added keepalive=20 to the scheduler Gerrit driver and restarted scheduler container # [[phab:T417497|T417497]] * 06:58 jeena: Updating development images on contint primary for [[phab:T415922|T415922]] === 2026-02-17 === * 23:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240081 * 23:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240078 * 15:58 brennen: deployed latest phab/phorge wmf/stable to devtools test instance ([[phab:T417657|T417657]]) * 09:01 hashar: Reloaded Zuul to enable php 8.5 testing on utfnormal, php-session-serializer, wikipeg, mediawiki/libs/Dodo, mediawiki/libs/UUID, testing-access-wrapper and translatewiki # [[phab:T406326|T406326]] === 2026-02-16 === * 15:27 hashar: Manually cleaned some old workspaces on integration-agent-docker-1042 === 2026-02-12 === * 20:07 James_F: Zuul: Enable PHP 8.5 jobs for most MW libraries, for [[phab:T406326|T406326]] * 19:33 James_F: Docker: [php83] Re-build with upstream's new 8.3.30 release and cascade * 19:31 James_F: Zuul: Add PHP 8.5 CI job to various things noted as blocked by Phan, for [[phab:T410941|T410941]], [[phab:T406326|T406326]] * 16:35 Krinkle: Disable publishing noise on tasks from repos Bcp47, clover-diff, ScopedCallback, and IDLeDOM. Ref [[phab:T143162|T143162]] * 15:53 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/87 * 11:21 James_F: Zuul: [mediawiki/libs/shellbox] Add direct Phan job, for [[phab:T416064|T416064]] === 2026-02-10 === * 20:16 dancy: Rebooted k3s.catalyst-dev (it was unresponsive, but the reboot hasn't helped) === 2026-02-09 === * 21:58 James_F: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job, for [[phab:T410941|T410941]] * 19:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1238006 [[phab:T415680|T415680]] * 11:51 James_F: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency, for [[phab:T414435|T414435]] === 2026-02-05 === * 17:58 James_F: Zuul: […/WikimediaCustomizations] Add six new dependencies for [[phab:T404334|T404334]] * 15:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1237254 * 15:18 James_F: Zuul: […/OATHAuth] Add dependency and phan dependency on CentralAuth === 2026-02-04 === * 12:54 James_F: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency * 10:03 hashar: Restarted Jenkins on releases2003.codfw.wmnet === 2026-02-02 === * 21:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1234926 "re-enable master jobs for some BlueSpice repos - [[phab:T403196|T403196]]" * 21:05 bd808: Unblock 85.146.0.0/17 ([[phab:T416079|T416079]]) * 19:47 James_F: Zuul: […/WikimediaCustomizations] Add cldr phan dependency, for [[phab:T404334|T404334]] * 17:33 bd808: Unblock 188.188.0.0/15 ([[phab:T416095|T416095]]) * 17:26 bd808: Unblock 85.94.84.0/22 ([[phab:T416105|T416105]]) * 17:09 bd808: Unblock 94.234.0.0/16 ([[phab:T416165|T416165]]) * 16:51 dancy: Update gitlab-runners to alpine-v18.6.6 ([[phab:T415214|T415214]]) * 16:27 bd808: Unblock 47.231.208.0/21 ([[phab:T416010|T416010]]) * 11:39 James_F: Zuul: […/WikimediaCustomizations] Add five new phan dependencies, for [[phab:T404334|T404334]] * 09:45 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 58532, 58557 === 2026-01-31 === * 21:49 James_F: Deleted Jenkins's job entry for castor-save-workspace-cache {{Gerrit|6193776}} and this seems to have unstuck things for [[phab:T416078|T416078]]? * 21:45 James_F: Running `sudo systemctl restart jenkins` on contint for [[phab:T416078|T416078]] * 21:44 James_F: Fighting [[phab:T416078|T416078]], took integration-castor-5 offline, disconnected, sshed in to kill threads, then reconnected; no change in aspect. * 19:03 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1235380 === 2026-01-28 === * 21:26 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn # [[phab:T415832|T415832]] * 21:11 bd808: Unblock 181.160.0.0/15 & 186.40.128.0/17 ([[phab:T415820|T415820]]) * 17:01 bd808: Unblock 102.182.0.0/16 ([[phab:T415782|T415782]]) === 2026-01-27 === * 16:45 James_F: Zuul: Switch skin-quibble template with identical extension-quibble, for [[phab:T402398|T402398]] * 16:18 James_F: Zuul: [ArticleGuidance] mention it will be in production * 15:55 James_F: Docker: [quibble-bullseye] Update to Quibble 1.15.0 * 15:12 James_F: Docker: [quibble-coverage] Pass PHPUnit config location explicitly, for [[phab:T395470|T395470]] * 09:18 hashar: integration: on integration-castor05, deleted caches for old MediaWiki branches * 09:15 hashar: integration: on pkgbuilder instances, removed Buster cow images, aptcache and hooks. `sudo cumin --force -p 0 'name:pkgbuilder' 'rm -fR /srv/pbuilder/<nowiki>{</nowiki>base-buster-amd64.cow,hooks/buster,aptcache/buster-amd64<nowiki>}</nowiki>'` # [[phab:T397209|T397209]] * 09:14 hashar: integration: cleaned up old workspaces under /srv/jenkins/workspace === 2026-01-26 === * 23:27 bd808: Unblock 66.130.0.0/15 ([[phab:T415596|T415596]]) * 22:52 bd808: Unblock 45.16.0.0/12 ([[phab:T415467|T415467]]) * 14:46 hashar: gerrit: changed `operations/software/permissions` project type from `CODE` to `PERMISSIONS` by pointing `HEAD` to `refs/meta/config` === 2026-01-22 === * 17:36 James_F: Docker: [quibble-coverage] Stop using legacy PHPUnit entrypoint ([[phab:T395470|T395470]]) & Stop excluding Dump/ParserFuzz/Stub groups ([[phab:T415230|T415230]]) * 15:11 James_F: Zuul: [mediawiki/extensions/Math] Add a standalone job, for [[phab:T415230|T415230]] === 2026-01-20 === * 20:38 bd808: Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229186 ([[phab:T415113|T415113]]) * 19:05 bd808: Rebooted deployment-cache-text08 to see if the mystery haproxy startup failure would go away ([[phab:T415100|T415100]]) * 18:50 bd808: Unblock 152.7.0.0/16 ([[phab:T415100|T415100]]) === 2026-01-17 === * 23:32 ori: beta-scap with `php_l10n: true` completed successfully: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/241466/console. PHP l10n files generated. Reverted local change to scap.cfg. * 23:26 ori: Temporarily set `php_l10n: true` on deployment-deploy04:/etc/scap.cfg to see if next scap succeeds. === 2026-01-16 === * 16:33 dancy: Deleting deployment-mx03.deployment-prep ([[phab:T412975|T412975]]) === 2026-01-15 === * 14:50 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/ArticleSummaries/ # [[phab:T413232|T413232]] === 2026-01-14 === * 17:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226907 * 16:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226893 * 15:57 bd808: Unblock 190.60.63.0/24 ([[phab:T414541|T414541]]) === 2026-01-13 === * 15:04 James_F: Zuul: Make quibble-for-mediawiki-core-vendor-mysql-php84 voting, for [[phab:T386108|T386108]] === 2026-01-12 === * 21:33 zabe: zabe@deployment-mwmaint03:~$ foreachwiki migrateLinksTable.php --table imagelinks # [[phab:T413668|T413668]] * 21:06 bd808: Unblock 66.81.168.0/21 ([[phab:T414303|T414303]]) * 17:42 dancy: Turned off instance deployment-prep.deployment-mx03 * 11:44 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 46331, 46344 === 2026-01-10 === * 21:48 taavi: reload zuul for https://gerrit.wikimedia.org/r/1224782 * 00:25 bd808: Unblock 91.160.0.0/12 ([[phab:T414190|T414190]]) === 2026-01-09 === * 17:33 thcipriani: re-enabling beta update jobs after test bad extension-list [[phab:T411516|T411516]] * 17:09 thcipriani: disabling beta update jobs to test bad extension-list [[phab:T411516|T411516]]) === 2026-01-08 === * 21:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224815 [[phab:T414136|T414136]] * 18:24 bd808: Unblock 89.80.0.0/12 ([[phab:T414113|T414113]]) * 15:55 dancy: Upgrading gitlab-runner to v18.5.0 on gitlab-cloud-runners. ([[phab:T414053|T414053]]) === 2026-01-07 === * 23:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1082574 https://gerrit.wikimedia.org/r/1224157 https://gerrit.wikimedia.org/r/1224159 * 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896311 [[phab:T27482|T27482]] * 23:06 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224218 * 17:34 James_F: Zuul: Add new extensions: IssueTrackerLinks, PreviewLinks, and WikiRAG * 17:34 James_F: Zuul: [labs/tools/heritage] Point to the task to drop 8.1 testing * 15:09 James_F: Zuul: [labs/tools/heritage] Add testing in PHP 8.2+, not just PHP 8.1 * 15:03 James_F: Zuul: Even for extension-broken, don't offer PHP 8.1 testing * 15:02 James_F: Zuul: Move quibble experimental sqlite/postgres tests to PHP 8.3 === 2026-01-06 === * 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223690 [[phab:T411814|T411814]] * 16:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223189 [[phab:T411814|T411814]] * 00:30 bd808: Unblock 85.134.128.0/17 ([[phab:T413755|T413755]]) * 00:02 bd808: Unblock 89.166.128.0/17 ([[phab:T413702|T413702]]) === 2026-01-05 === * 23:57 bd808: Unblock 185.233.104.0/22 ([[phab:T413472|T413472]]) * 23:51 bd808: Unblock 45.62.112.0/21 ([[phab:T413079|T413079]]) * 23:44 bd808: Unblock 85.134.200.0/21 ([[phab:T413067|T413067]]) * 19:03 dancy: Updated buildkitd to v0.26.3 in gitlab-cloud-runners * 14:27 taavi: reload zuul for {{Gerrit|1223191}} * 13:57 James_F: Zuul: [mediawiki/php/wmerrors] Enable PHP 8.5 testing, for [[phab:T410921|T410921]] === 2026-01-03 === * 17:59 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222709 https://gerrit.wikimedia.org/r/1220388 https://gerrit.wikimedia.org/r/1219140 === 2026-01-02 === * 17:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222597 === 2026-01-01 === * 02:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1221644 <noinclude>'''Server Admin Log''' logged from {{IRC|wikimedia-releng}} for [[Nova Resource:Deployment-prep|Beta Cluster]], [[mw:Continuous integration|Continuous integration]] and various other Release Engineering projects.</noinclude> {{SAL-archives/Release Engineering}} <noinclude>[[Category:SAL]]</noinclude> frpllpj5q49tblwazn2i4lexvbgakc5 2452817 2452790 2026-08-31T09:16:34Z Stashbot 7414 elukey: move cx-server and citoid-beta endpoints in deployment-prep to two new Trixie VMs, update their configs and Docker images and delete the old images. 2452817 wikitext text/x-wiki === 2026-08-31 === * 09:16 elukey: move cx-server and citoid-beta endpoints in deployment-prep to two new Trixie VMs, update their configs and Docker images and delete the old images. * 04:18 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1332530 === 2026-08-27 === * 15:06 Krinkle: Disable duplicate publishing noise from extension-IPReputation, EIPR, [[phab:T143162|T143162]] === 2026-08-26 === * 18:07 andrewbogott: resolving rebase conflicts in /srv/git/labs/private * 17:24 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1329617 * 14:45 hashar: Updating Castor (0.4.3..0.4.5) on all Jenkins jobs {{!}} https://gerrit.wikimedia.org/r/1329577 * 07:11 hashar: integration: updated Quibble jobs to replace deprecated `--commands` option by 1..n `--command` option(s) {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1293711 {{!}} [[phab:T321617|T321617]] === 2026-08-25 === * 21:32 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 * 21:23 brett: revert deployment-prep acme-chief switch: acme-chief active to deployment-acme-chief05 and passive to deployment-acme-chief06 * 21:03 brett: Switch acme-chief active to deployment-acme-chief07, passive to deployment-acme-chief08 * 20:31 Southparkfan: [[phab:T401839|T401839]] - provisioned deployment-docker-wikifeeds01 (trixie) using Tofu (cloudvps-repos/deployment-prep/tofu-provisioning) * 15:06 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/121 ([[phab:T435368|T435368]]) * 07:35 taavi@cloudcumin1001: END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance deployment-ircd03 * 07:35 taavi@cloudcumin1001: START - Cookbook wmcs.vps.remove_instance for instance deployment-ircd03 === 2026-08-24 === * 08:29 hashar: integration: add firewall rule to ssh from contint1003/contint2003 IPv6 (IPv4 was already allowed) # [[phab:T435756|T435756]] * 08:26 hashar: integration: removing firewall rule for ssh from contint1002/contint2002 IPv4 # [[phab:T418521|T418521]] === 2026-08-21 === * 13:07 hashar: integration: triggered Doxygen doc for PersonalDashboard extension using: `zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/PersonalDashboard --change {{Gerrit|1328044}},1` # [[phab:T435392|T435392]] === 2026-08-20 === * 23:26 bd808: deployment-mediawiki14: `systemctl stop php8.3-fpm; sleep 5m; systemctl start php8.3-fpm` -- maybe the bot storm will break if we give fast 500 responses for 5 minutes. * 14:55 James_F: jforrester@integration-castor06:$ sudo rm -rf /srv/castor/castor-mw-ext-and-skins/master/wikilambda-catalyst-end-to-end # Clear stale Catalyst castor npm downloads. * 13:35 James_F: Zuul: [mediawiki/extensions/WikiLambda] Re-enable Catalyst * 08:54 hashar: deployment-prep: hard reboot deployment-cache-test-08 # [[phab:T435421|T435421]] === 2026-08-19 === * 14:48 hashar: integration: updated castor save job to have rsync emit statistics {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1321049 {{!}} [[phab:T432685|T432685]] * 12:30 James_F: Updating development images on contint primary for “fundraising: Drop libc-client-dev from the bookworm PHP 8.2 image” === 2026-08-18 === * 17:32 dancy: Rebooting deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud for good measure * 17:31 dancy: rm /var/log/apache2/*.gz on deployment-mediawiki14.deployment-prep.eqiad1.wikimedia.cloud to free up ~6GB. * 09:24 hashar: zuul: restarted zuul-web === 2026-08-17 === * 21:09 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/115 * 21:01 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/118 ([[phab:T413817|T413817]], [[phab:T407430|T407430]]) * 18:08 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/113 * 16:30 James_F: Zuul: [mediawiki/extensions/WP25EasterEggs] Archive repository, for [[phab:T418134|T418134]] === 2026-08-14 === * 18:59 James_F: Zuul: Enforce CI for mediawiki-php-<nowiki>{</nowiki>excimer,luasandbox,wikidiff2<nowiki>}</nowiki> for [[phab:T425943|T425943]] * 18:20 James_F: Docker: [php85] Migrate to Wikimedia-provide binary, cascaded, for [[phab:T433254|T433254]] * 14:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1325900 === 2026-08-13 === * 19:55 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 ([[phab:T401115|T401115]]) (redux for typo fix) * 19:27 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/111 ([[phab:T401115|T401115]]) * 03:05 TimStarling: created Produnto tables on beta [[phab:T421436|T421436]] === 2026-08-12 === * 12:31 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/mwext-phpunit-coverage*/npm` # [[phab:T427922|T427922]] * 11:59 hashar: integration: on Castor: `sudo rm -fR /srv/castor/*/*/*codehealth*/npm` # [[phab:T427822|T427822]] === 2026-08-07 === * 16:13 hashar: integration: deleted integration-agent-[[phab:T422258|T422258]] agent # [[phab:T422258|T422258]] * 14:31 hashar: integration: updating Quibble jobs to Quibble 1.19.0 # [[phab:T432934|T432934]] [[phab:T432943|T432943]] [[phab:T427922|T427922]] * 07:25 hashar: Tag Quibble 1.19.0 @ {{Gerrit|a8a84ed1c0adb34688ce22066ebde8a2d584a7d4}} # [[phab:T432934|T432934]] [[phab:T432943|T432943]] [[phab:T427922|T427922]] === 2026-08-06 === * 18:02 dancy: Restarted gitlab-webhooks ([[phab:T430410|T430410]]) (revert) * 17:55 dancy: Restarted gitlab-webhooks ([[phab:T430410|T430410]]) === 2026-08-05 === * 20:31 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1184176 * 18:39 James_F: Docker: [php85] Upgrade PHP to 8.5.9 * 15:58 James_F: Zuul: [mediawiki/extensions/WikiLambda] Disable Catalyst, corrupted npm cache === 2026-08-04 === * 19:06 dancy: Updating buildkitd to v0.32.2 on gitlab-cloud-runners (staging and production) ([[phab:T433985|T433985]]) * 16:48 James_F: Docker: [php83] Upgrade PHP to 8.3.33 === 2026-08-03 === * 20:45 dancy: Updating buildkitd to v0.32.1 on gitlab-cloud-runners (staging and production) ([[phab:T433879|T433879]]) * 16:40 hashar: gerrit: added Vaughn Walters to integration group until he get added to the ciadmin LDAP group {{!}} [[phab:T433615|T433615]] * 16:37 James_F: Zuul: Drop REL1_44 testing, EOL, for [[phab:T428911|T428911]] === 2026-07-31 === * 18:14 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/114 * 12:23 hashar: integration: sudo cumin --force -p 0 'name:docker' 'rm -fR /srv/jenkins/workspace/*pipeline*' === 2026-07-30 === * 17:28 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # [[phab:T424419|T424419]] === 2026-07-29 === * 21:33 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/109 * 18:33 dancy: Buildkit v0.32.0 deployed to gitlab-cloud-runners staging and production ([[phab:T433520|T433520]]) === 2026-07-24 === * 17:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1316011 === 2026-07-23 === * 15:06 dancy: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/1314124 ([[phab:T295351|T295351]]) === 2026-07-22 === * 21:32 dancy: Moved /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/npm to /srv/castor-debug-[[phab:T20260722|T20260722]]-npm-torn-cacache/ on integration-castor06 * 18:41 dancy: Zuul dependencies upgraded and Zuul restarted. * 18:25 dancy: Zuul is currently broken due to the Gerrit SSH key update. I'm investigating * 17:24 brennen: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1314014/1 ([[phab:T432886|T432886]]) * 16:58 dancy: Restarting Gerrit ([[phab:T240266|T240266]]) (again) * 15:59 dancy: Restarting Gerrit ([[phab:T240266|T240266]]) * 09:54 James_F: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure for {{Gerrit|1313321}}, per Lucas_WMDE. === 2026-07-21 === * 15:59 dancy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1313220 * 07:08 hashar: integration: cleaned up Jenkins workspace on integration-agent-docker-1084 === 2026-07-20 === * 19:36 dancy: Restarting jenkins on contint1003 to clear out hours-long stuck jobs === 2026-07-16 === * 23:07 mutante: gerrit1003/2002/2003: rm /srv/gerrit/.ssh/config and revert gerrit:1311577 which fixed [[phab:T398401|T398401]] but caused [[phab:T432413|T432413]] - replication is working again * 18:51 dancy: Updated buildkit to v0.31.2 gitlab-cloud-runners (staging and production) ([[phab:T432360|T432360]]) * 18:37 dancy: Restarted Jenkins to unstick jobs. * 18:33 dancy: Jenkins placed in shutdown mode in preparation for a restart * 17:07 bd808: Hard reboot of deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud via Horizon; console shows OOM ([[phab:T432374|T432374]]) * 16:40 brennen: contint1002: chown and chmod on /srv/zuul/git/mediawiki/extensions/CampaignEvents/.git per https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions * 16:34 dancy: Restarted docker on contint2003 ([[phab:T432326|T432326]]) * 16:33 dancy: Restarted docker on contint1003 ([[phab:T432326|T432326]]) * 16:32 dancy: Restarted docker on contint1003 * 12:31 Krinkle: krinkle@contint1003:~$ sudo /usr/sbin/service jenkins restart * 12:10 Krinkle: krinkle@contint1002: zuul restart === 2026-07-15 === * 18:58 James_F: dev-images: Re-build PHP images for latest point releases, for [[phab:T431099|T431099]] * 09:13 James_F: Docker: [php83] Update PHP to 8.3.32 for [[phab:T431099|T431099]] === 2026-07-14 === * 11:53 James_F: Docker: [commit-message-validator] Update to v3.0.0, for [[phab:T431799|T431799]] * 08:37 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud === 2026-07-13 === * 15:03 dancy: Updating gitlab-cloud-runners to v19.0.2 === 2026-07-10 === * 11:56 James_F: Zuul: Disable all browser tests on release branches except Wikibase's, for [[phab:T430415|T430415]] * 09:16 James_F: Docker: [commit-message-validator] Update to 2.3.0 === 2026-07-09 === * 11:55 hashar: retriggering postmerge change for [[phab:T431582|T431582]]: zuul enqueue --trigger gerrit --pipeline postmerge --project machinelearning/liftwing/inference-services --change {{Gerrit|1308631}},3 === 2026-07-08 === * 20:58 brennen: patchdemo: deployed https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/367 ([[phab:T427964|T427964]]) * 19:21 mutante: gerrit - replacing registerEmailPrivateKey in Gerrit config - this invalidates pending/outstanding email validation links for gerrit users - but does not affect active accounts or already verified email addresses * 14:36 hashar: contint1003, contint2003: manually installed `docker-buildx` Debian package to validate https://gerrit.wikimedia.org/r/c/operations/puppet/+/1308659 # [[phab:T431582|T431582]] * 13:04 hashar: deployment-prep: git repack on /srv/mediawiki-staging/php-master * 12:27 hashar: deployment-prep: on deployment server: clearing old branches for mediawiki/extensions and mediawiki/skins # [[phab:T428864|T428864]] === 2026-07-07 === * 19:41 mutante: contint1003/2003 - add jenkins-agent user to docker group; restart jenkins * 17:17 hashar: gerrit: deleted /srv/gerrit/java_pid3571660.hprof === 2026-07-06 === * 20:06 dancy: Updated buildkitd to v0.31.1 in gitlab-cloud-runners ([[phab:T429988|T429988]]) * 08:11 James_F: Zuul: Add WikimediaAntiAbuse extension, for [[phab:T431023|T431023]] === 2026-07-03 === * 15:25 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add Elastica dep too * 15:09 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CirrusSearch dep * 11:22 James_F: Zuul: Make the in-mediawiki-tarball template real * 10:06 James_F: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches * 09:53 James_F: Docker: [quibble-coverage] Update phpunit-patch-coverage to 0.0.18, for [[phab:T423987|T423987]] and [[phab:T425807|T425807]] === 2026-07-02 === * 13:27 James_F: Docker: [composer-scratch] Upgrade composer to 2.10.2 and cascade, for [[phab:T428570|T428570]] * 10:47 hashar: zuul1002: running Puppet agent to drop `wikimediacloud.org` from `no_proxy` {{!}} [[phab:T430479|T430479]] === 2026-07-01 === * 13:39 hashar: integration: added timestamping to operations-puppet-catalog-compiler and operations-puppet-catalog-compiler-puppet7-test jobs * 02:45 hashar: gerrit: on gerrit2003 deleted /srv/gerrit/java_pid3520115.hprof (the JVM apparently died at some point, I assume due to heavy crawling) === 2026-06-30 === * 15:13 dancy: Rebooting deployment-mwlog02.deployment-prep to clear stuck udp2log processes === 2026-06-29 === * 09:32 hashar: gerrit: deleted repository phabricator/extensions/BurnDownCharts , created in July 2014, had no commit/changes === 2026-06-28 === * 15:30 hashar: Updated integration/zuul-jobs from upstream (c75fe6ef19c..fc4af6d4471), notably to remove `requestsexceptions` in `upload-logs-swift` role # [[phab:T430458|T430458]] === 2026-06-26 === * 13:59 Krinkle: [[phab:T429658|T429658]] krinkle@doc1004:/srv/doc/cover-extensions$ sudo -u doc-uploader rm -rf ShortUrl/ * 13:59 Krinkle: [[phab:T429658|T429658]] krinkle@doc2003:/srv/doc/cover-extensions$ sudo -u doc-uploader rm -rf ShortUrl/ === 2026-06-25 === * 20:57 dancy: Restarting Jenkins to unstick builds * 20:49 dancy: Investigating castor-save-workspace-cache clog * 16:50 inflatador: add 60GB cinder vol to deployment-cirrussearch15 [[phab:T425585|T425585]] * 14:25 inflatador: delete unused servers deployment-cirrussearch1[2-4] [[phab:T425585|T425585]] * 10:07 James_F: Docker: Bump Node 24 / Node 26 to new releases === 2026-06-24 === * 17:26 dancy: Set `profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py` in hiera config for deployment-puppetserver prefix ([[phab:T429413|T429413]]) * 15:12 dancy: sudo systemctl restart php8.3-fpm on deployment-jobrunner05 (Attempting to resolve logspam) === 2026-06-23 === * 14:31 brennen: deploying patchdemo for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/361 === 2026-06-22 === * 23:19 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/mediawiki-core/master/mediawiki-node24/ #[[phab:T429824|T429824]] * 23:07 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ #[[phab:T429824|T429824]] (again) * 22:35 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ #[[phab:T429824|T429824]] * 19:30 thcipriani: thcipriani@integration-castor06:~$ sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-with-gated-extensions-vendor-mysql-php83 #[[phab:T429824|T429824]] === 2026-06-19 === * 20:13 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1304632 * 09:39 Lucas_WMDE: deployment-deploy04: used createAndPromote to restore User:Lucas Werkmeister (WMDE) to bureaucrat (dewiki, enwiki, metawiki, wikidatawiki) and wikidata-staff (wikidatawiki) after groups were very unhelpfully removed, presumably due to 2FA enforcement, without apparent warning or announcement of any kind * 07:07 dcausse: restarted php-fpm on deployment-jobrunner05 (inconsistent php state: MediaWiki\Extension\PageAssessments\HookHandler\ParserHooks::__construct(): Argument #3 ($config) must be of type MediaWiki\Config\Config, MediaWiki\Page\WikiPageFactory given) * 06:13 hashar: Upgrading Quibble jobs to 1.18.2 (retry git connections on reset) https://gerrit.wikimedia.org/r/c/integration/config/+/1304181 # [[phab:T420865|T420865]] === 2026-06-18 === * 19:52 dancy: Building quibble 1.18.2 images on contint primary ([[phab:T420865|T420865]]) * 18:50 dancy: Tag Quibble 1.18.2 @ {{Gerrit|152b497d317eaafc3cec334d5ce7e549697a2980}} # [[phab:T420865|T420865]] * 15:15 dancy: Deleting deployment-db11 and deployment-db14 ([[phab:T428910|T428910]]) * 07:23 dcausse: reindexing all wikis to opensearch2 ([[phab:T425585|T425585]], [[phab:T427196|T427196]]) === 2026-06-17 === * 23:16 dduvall: restarted zuul to clear up 5 hrs of stuck queues * 23:03 mutante: re-enabled puppet on contint1003 - triple checked puppet does NOT start jenkins anymore. BOTH masked AND stopped while running untouched on contint1002. after: gerrit:1303578 {{!}} ([[phab:T418521|T418521]]) ([[phab:T428791|T428791]]) * 18:51 dduvall: performed stop/start of jenkins service on contint1002 following failed safeRestart * 18:45 dduvall: restarting jenkins due to stuck zuul queues * 14:56 inflatador: mwscript /srv/mediawiki-staging/php-master/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki=enwikibooks === 2026-06-16 === * 16:00 dancy: sudo keyholder arm on deployment-deploy04.deployment-prep * 15:48 dancy: Resizing deployment-deploy04.deployment-prep from g4.cores4.ram8.disk20 to g4.cores8.ram16.disk20 ([[phab:T429364|T429364]]) * 15:31 dancy: Turning off deployment-db11 and deployment-db14 === 2026-06-15 === * 23:40 dancy: systemctl restart php8.3-fpm on deployment-jobrunner05 to reload beta db configuration ([[phab:T428930|T428930]]) * 21:00 dancy: deployment-db15 promoted to master ([[phab:T428930|T428930]]) * 20:53 dancy: deployment-db11.deployment-prep going read-only * 20:48 dancy: deployment-db11.deployment-prep will be going read-only soon while db15 is being promoted to primary. * 20:26 dancy: Added deployment-db16 ([[phab:T429245|T429245]]) * 18:26 dancy: Rebooting deployment-jobrunner05 === 2026-06-12 === * 21:45 dancy: Unstuck wmf-beta-update-all service on deployment-deploy04.deployment-prep (sudo systemctl stop wmf-beta-update-all) * 18:00 thcipriani: unmasking jenkins on contint1002 and restarting * 17:49 thcipriani: attempting to cancel castor-save-workspace-cache {{Gerrit|6710545}} * 15:19 James_F: Docker: [php83] Re-platform to Debian Bookworm, for [[phab:T383337|T383337]] * 15:07 dancy: deployment-db15 configured as a replica of deployment-db11 ([[phab:T428930|T428930]]) * 10:21 Krinkle: `krinkle@<nowiki>{</nowiki>doc1004,doc2003<nowiki>}</nowiki>:/srv/doc/mediawiki-core$ sudo -u doc-uploader rm -rf list/` - remove doc build for git-tag test. === 2026-06-11 === * 14:46 hashar: for minor in $(seq 21 42); do ./.tox/make-release/bin/python -u ./make-release/branch.py --delete --abandon --bundle '*' "REL1_$minor"; done; * 14:46 hashar: On all MediaWiki repos, converting old release branches up to REL1_42 included to tags. Last time I missed non wmf repo # [[phab:T380841|T380841]] {{!}} [[phab:T428864|T428864]] * 09:45 hashar: Converted mediawiki/core branches REL1_39, REL1_40, REL1_41, REL1_42 to tags # [[phab:T428864|T428864]] * 09:18 hashar: Converting REL1_42 branches to tags # [[phab:T428864|T428864]] * 09:18 hashar: Converting REL1_41 branches to tags # [[phab:T428864|T428864]] * 09:05 hashar: Converting REL1_40 branches to tags # [[phab:T428864|T428864]] * 08:56 hashar: Converting REL1_39 branches to tags # [[phab:T428864|T428864]] * 08:40 hashar: gerrit: deleted mediawiki/core branch "development" that pointed to {{Gerrit|7f622781cc31053b121f6f4ddbff506cba10d38e}} (which is contained by master). Had probably been created by a direct push. === 2026-06-10 === * 16:06 James_F: Docker: Provide quibble-bookworm, for [[phab:T362705|T362705]] === 2026-06-04 === * 23:39 jeena: Updating development images on contint primary for [[phab:T424691|T424691]] * 12:30 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc # fix failure seen in mwext-node24-rundoc 4812 * 08:35 hashar: Built Docker images `docker-registry.wikimedia.org/releng/java21:0.1` and `docker-registry.wikimedia.org/releng/maven-java21:0.1` # [[phab:T412978|T412978]] === 2026-06-03 === * 14:19 jnuche: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/107 * 12:54 James_F: Zuul: Add Rae 5e as a trusted user * 08:26 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1296559 "inference-services: Add LLM generated editing suggestions CI/CD pipelines." # [[phab:T427794|T427794]] === 2026-06-02 === * 23:23 thcipriani: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1296695 * 23:10 thcipriani: tag quibble 1.18.1 @ {{Gerrit|4b7959553c095d811426f394165c69ecc13a44eb}} * 18:13 brennen: devtools phab/phorge: deployed work/2026-06-01-merge-phorge to https://phabricator.wmcloud.org/ for testing ([[phab:T410849|T410849]]) * 16:05 hashar: integration: upgraded pypy from 7.3.11 (py3.9) to 7.3.20 (py3.11) # [[phab:T423607|T423607]] * 15:49 jnuche: Updating buildkitd to v0.30.0 in gitlab-cloud-runners ([[phab:T426212|T426212]]) * 15:24 hashar: Building docker images for https://gerrit.wikimedia.org/r/c/integration/config/+/1295865 # [[phab:T423607|T423607]] * 14:57 jnuche: Jenkins/Zuul is back * 14:38 jnuche: restarting Jenkins * 14:28 jnuche: bring back castor node, that didn't help * 14:23 jnuche: trying to reconnect castor node, see if that helps somehow * 14:12 jnuche: option to "Enable Gearman" times out. Can't re-enable from UI. Gearman plugin logs are empty. Neat * 14:05 jnuche: trying to reconnect Gearman === 2026-06-01 === * 21:34 jeena: Updating development images on contint primary for [[phab:T424663|T424663]] * 10:58 hashar: gerrit: flushed `ldap_usernames` cache in case a missing account ended up being cached there # [[phab:T427792|T427792]] === 2026-05-28 === * 16:48 hashar: castor: nuked SonarQube cache: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-codehealth-master-non-voting/sonar/ # [[phab:T427471|T427471]] * 07:05 hashar: integration: delete deployment-deploy04 agent from Jenkins controller. Batch job has been migrated to a systemd driven script # [[phab:T256168|T256168]] * 06:29 hashar: integration: delete integration-castor05 agent from Jenkins, replaced by integration-castor06 # [[phab:T421114|T421114]] === 2026-05-27 === * 21:07 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1294425 * 09:49 hashar: Updated helm-lint Jenkins job to use releng/helm-linter:0.8.0 image # [[phab:T424824|T424824]] * 08:25 codders: integration-castor06: sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ === 2026-05-26 === * 15:26 James_F: Zuul: Add PHP 8.5 to a few missing PHP pipelines, oops * 14:46 hashar: Updating Quibble Jenkins jobs to stop Supervisord from spawning Memcached # [[phab:T397810|T397810]] * 14:07 hashar: Updated integration-quibble-* jobs in order to validate running tests with Supervisord not managing memcached # [[phab:T397810|T397810]] * 13:15 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 981, 985, 987 * 09:05 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1292620 (Introduce Phan composer job - [[phab:T231966|T231966]]) === 2026-05-21 === * 16:33 hashar: Reloaded Zuul to enable Node24 CI job for `labs/tools/wdaudiolex-fe` # [[phab:T426366|T426366]] === 2026-05-19 === * 18:07 James_F: Zuul: [mediawiki/libs/ZestJQ] Add basic PHP and Node CI === 2026-05-18 === * 21:12 James_F: Zuul: [operations/software/gerrit] Add Node26 as experimental * 21:10 mutante: gerrit-replica.wikimedia.org, gerrit-spare.wikimedia.org - rebooting backends * 20:57 James_F: Zuul: [integration/docroot] Test in PHP 8.3+, dropping 8.2 * 20:56 James_F: Zuul: [analytics/wmde/scripts] Test in PHP 8.3+, dropping 8. * 20:18 James_F: Zuul: [wikimedia/fundraising/dash] Replace Node 20 testing with Node 24 * 20:18 James_F: Zuul: Migrate various labs things to Node 24 * 20:02 James_F: Docker: [ajv, sonar-scanner] Migrate to Node 24 * 19:58 James_F: Zuul: Migrate various production/CI things to Node 24 * 18:17 mutante: releases.wikimedia.org - rebooting backends * 18:14 mutante: rebooting production gitlab-runners * 18:12 dancy: gitlab-cloud-runners have been revived. * 18:11 James_F: Zuul: [design/codex] Switch CI to Node 24 * 15:52 dancy: gitlab-cloud-runners are in a broken state. I'm investigating * 14:27 hashar: Upgrading Quibble jobs to 1.18.0 * 09:29 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 22260 === 2026-05-15 === * 18:36 dancy: Upgraded gitlab-cloud-runners (prod) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 18:24 dancy: Upgrading gitlab-cloud-runners (prod) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 18:11 dancy: Upgraded gitlab-cloud-runners (staging) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 17:59 dancy: Upgrading gitlab-cloud-runners (staging) from 1.35.1-do.5 to 1.35.1-do.6 ([[phab:T426436|T426436]]) * 13:02 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1287828 [[phab:T426392|T426392]] === 2026-05-13 === * 12:42 James_F: Zuul: [mediawiki/extensions/Springboard] Add AdminLinks Phan dependency * 12:42 James_F: Zuul: [mediawiki/extensions/ChatBot] Add dependencies on VisualEditor and BlueSpiceFoundation * 12:42 James_F: Zuul: [mediawiki/extensions/ChatIntegration] Add dependency on VisualEditor * 12:37 James_F: Zuul: [mediawiki/extensions/WikiLambda] Drop AF and SB deps down to phan-only, for [[phab:T423180|T423180]] === 2026-05-12 === * 20:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/104 ([[phab:T424774|T424774]]) * 18:08 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add AF and SB deps for [[phab:T423180|T423180]] * 14:18 atsukoito: PrivateSettings: empty $wgOpensearchCredentials for opensearch-on-k8s synced to deploy04 by Reedy * 13:04 atsukoito: PrivateSettings: credentials for opensearch-on-k8s ttmserver-test * 11:50 James_F: Zuul: [machinelearning/liftwing/inference-services] Add qwen36 llm model CI/CD pipelines, for [[phab:T425680|T425680]] * 11:46 James_F: Zuul: Add experimental php-pie-build* jobs to other PHP extensions, for [[phab:T425943|T425943]] * 11:37 James_F: Zuul: [mediawiki/php/wikidiff2] Add experimental php-pie-build* jobs, for [[phab:T425943|T425943]] * 10:05 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 # fix failure seen in quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 7817 * 08:44 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51633 === 2026-05-11 === * 18:28 James_F: Docker: Add changes to php-compile images for PIE, for [[phab:T425943|T425943]] * 16:06 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51439 and 51452 === 2026-05-09 === * 20:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1285498 === 2026-05-07 === * 22:53 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/105 === 2026-05-06 === * 18:13 bd808: Unblock 88.165.192.0/19 * 18:03 bd808: Unblock 94.208.0.0/14 * 17:56 bd808: Unblock 84.226.0.0/16 * 17:41 bd808: Unblock 94.34.0.0/16 * 17:35 bd808: Unblock 109.134.0.0/16 === 2026-05-05 === * 21:20 James_F: Zuul: Provide Node 26 experimental jobs everywhere needed * 21:04 James_F: Docker: Provide initial Node 26 images * 19:01 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto dependency, for [[phab:T396135|T396135]] * 14:58 dancy: rm /var/log/<nowiki>{</nowiki>user.log.1,syslog.1,messages.1<nowiki>}</nowiki> on deployment-eventgate-4.deployment- prep ([[phab:T425429|T425429]]) === 2026-05-04 === * 15:19 dancy: Upgrading gitlab cloud runners (prod) from 1.35.1-do.3 to 1.35.1-do.5 * 14:51 dancy: Upgrading gitlab cloud runners (staging) from 1.35.1-do.3 to 1.35.1-do.5 * 10:40 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template === 2026-05-02 === * 20:49 James_F: Zuul: [mediawiki/core] Enforce PHP 8.4 & 8.5 on release branches, all pass * 19:27 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for MW release branches * 19:19 James_F: Zuul: [mediawiki/extensions/BlogPage] Add dependencies * 16:48 James_F: Hard-restarting Zuul to clear the huge number of i18n updates being re-submitted. * 15:48 James_F: Zuul: [wikimedia-cz/*] Test in PHP 8.3+, dropping 8.2 * 14:02 TheresNoTime: Add bvibber to deployment-prep project * 09:08 James_F: Docker: [quibble-*] Add php-luasandbox so we can test both modes in Scribunto === 2026-05-01 === * 15:42 James_F: Zuul: [wikimedia/lucene-explain-parser] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: Zuul: [wikimedia/textcat] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: Zuul: [mediawiki/tools/ParseWiki] Test in PHP 8.3+, dropping 8.2 * 15:42 James_F: zuul: Add ToprakM to CI allowlist * 15:19 James_F: Zuul: [translatewiki] Test in PHP 8.3+, dropping 8.2 * 15:10 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add TestKitchen as a dependency, for [[phab:T425076|T425076]] * 12:40 James_F: Zuul: [mediawiki/tools/code-utils] Test in PHP 8.3+, dropping 8.2 * 08:02 James_F: Zuul: Update xtex's e-mail in the allowlist * 07:37 James_F: Zuul: Switch release branches' selenium jobs to PHP 8.3 * 07:33 James_F: Zuul: Test Wikimedia production libraries in PHP 8.3+, dropping 8.2 === 2026-04-30 === * 21:36 brennen: gitlab-webhooks: building & restarting to deploy https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40 * 20:26 James_F: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting * 20:16 James_F: jforrester@doc1004:~$ # sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn/ # [[phab:T415832|T415832]] * 20:14 James_F: Zuul: [mediawiki/extensions/WebAuthn] Archive, for [[phab:T415832|T415832]] / [[phab:T303495|T303495]] * 17:16 brennen: wikibugs: most maintainers at hackathon, so go release-engineering added as a maintainer while looking to debug error at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/810904 * 15:19 mutante: upgrading zuul to 14.2.0-1 on "new zuul" machines ([[phab:T424879|T424879]]) === 2026-04-29 === * 15:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add ConfirmEdit dependency, for [[phab:T424597|T424597]] * 15:36 James_F: Zuul: Drop experimental node22 jobs, never used in practice * 15:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1279392, https://gerrit.wikimedia.org/r/1279397 === 2026-04-28 === * 18:11 bd808: Unblock 86.0.0.0/16 * 17:41 bd808: Unblock 79.192.0.0/10 * 17:07 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing * 17:07 James_F: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing * 17:07 James_F: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing * 16:32 James_F: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing * 13:34 James_F: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing * 13:34 James_F: Zuul: [oojs/ui] Drop PHP 8.2 testing * 13:14 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI * 10:40 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717 * 00:03 bd808: Increase parallelism for wmf-beta-update-databases.py ([[phab:T256168|T256168]]) === 2026-04-27 === * 22:11 bd808: Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ ([[phab:T256168|T256168]]) * 21:57 bd808: Add web security group to deployment-deploy04 ([[phab:T256168|T256168]]) * 20:45 James_F: Zuul: Restrict mw*-codehealth-patch jobs to master only, for [[phab:T424573|T424573]] * 17:16 James_F: Docker: [mediawiki-phan-taint-check-demo] Re-platform to Trixie and so PHP 8.4 * 15:53 James_F: Zuul: [mediawiki/extensions/ReportIncident] Add TestKitchen phan dependency, for [[phab:T424220|T424220]] * 14:32 James_F: Zuul: Drop PHP 8.2 enforcement from MediaWiki things for master and REL1_46 for [[phab:T358667|T358667]] * 12:38 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 383 * 09:18 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # [[phab:T424419|T424419]] === 2026-04-26 === * 20:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276777 === 2026-04-24 === * 22:48 dduvall: merged zuul3 branch of integration/config into master and pushed (in preparation for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198) * 12:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276428 === 2026-04-23 === * 23:57 bd808: Set `profile::beta::autoupdater::run_updater: true` for deployment-deploy04 via Horizon ([[phab:T256168|T256168]]) * 22:58 bd808: bd808@deployment-deploy04 `sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-all` * 22:36 bd808: bd808@deployment-deploy04 `sudo -u mwdeploy /usr/local/bin/wmf-beta-update-all` * 22:16 bd808: Disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad so that replacement script can be tested ([[phab:T256168|T256168]]) * 22:12 bd808: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad so that replacement script can be tested ([[phab:T256168|T256168]]) * 22:02 bd808: Cherry-picked {{gerrit|1276813}} to deployment-puppetserver-1 ([[phab:T256168|T256168]]) * 20:11 James_F: Zuul: [wikibase/*] Replace CI testing in Node 20 with Node 24 * 20:11 James_F: Zuul: [wikidata/query/*] Replace CI testing in Node 20 with Node 24 * 20:11 James_F: Zuul: [analytics/*] Replace CI testing in Node 20 with Node 24 * 20:10 James_F: Zuul: [mediawiki/tools/*] Replace CI testing in Node 20 with Node 24 * 20:06 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]]) * 19:55 James_F: Zuul: [jquery-client] Replace CI testing in Node 20 with Node 24 * 19:51 James_F: Zuul: [wikipeg] Drop testing in Node 20 and Node 22 * 19:47 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.34.5-do.3 to 1.35.1-do.3 ([[phab:T423726|T423726]]) * 19:37 James_F: Zuul: [oojs/ui] Drop CI testing in Node 20 and Node 22 * 19:37 James_F: Zuul: [oojs/js] Drop CI testing in Node 20 and Node 22 * 19:37 James_F: Zuul: [unicodejs] Replace CI testing in Node 20 with Node 24 * 19:36 James_F: Zuul: [wikimedia/portals] Drop CI testing in Node 20 and Node 22 * 18:57 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]]) * 18:39 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.3 to 1.34.5-do.3 ([[phab:T423726|T423726]]) * 18:18 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]]) * 17:58 James_F: Zuul: [mediawiki/extensions/OAuth] Add dependency on CentralAuth, for [[phab:T415281|T415281]] * 17:56 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.13-do.2 to 1.33.9-do.3 ([[phab:T423726|T423726]]) * 16:35 James_F: Zuul: Enforce PHP 8.5 CI for MW things in master (and REL1_46), for [[phab:T411814|T411814]] * 16:19 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.5 CI * 15:47 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add AntiSpoof dependency, for [[phab:T420548|T420548]] * 14:20 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 8385 and 8405 * 12:56 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add CentralNotice dependency, for [[phab:T422082|T422082]] === 2026-04-22 === * 00:07 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency, for [[phab:T424113|T424113]] === 2026-04-21 === * 23:26 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too, for [[phab:T394410|T394410]] * 23:17 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs, for [[phab:T422031|T422031]] * 20:47 inflatador: updating cirrussearch hosts to Trixie/OpenSearch 2 [[phab:T421763|T421763]] * 20:38 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep, for [[phab:T394410|T394410]] * 20:17 bd808: Running tofu for [[phab:T421244|T421244]] * 18:00 James_F: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency * 16:35 bd808: Unblock 79.116.0.0/16 * 13:34 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep, for [[phab:T415254|T415254]] * 13:27 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency, for [[phab:T420548|T420548]] === 2026-04-20 === * 23:56 bd808: Unblock 76.157.0.0/16 * 18:28 dancy: Upgrading gitlab cloud runners (staging) to 1.33.9-do.2 ([[phab:T423726|T423726]]) * 18:28 dancy: Upgrading gitlab cloud runners (staging) ([[phab:T423726|T423726]]) * 18:19 James_F: jjb: All 486 (!) jobs now updated for [[phab:T423622|T423622]] * 18:18 bd808: Unblock 113.128.0.0/15 * 15:03 James_F: Docker: Bump ci-bullseye/-bookworm/-trixie for mirrors.wm.org removal, [[phab:T423622|T423622]] === 2026-04-19 === * 19:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272752 === 2026-04-17 === * 21:07 thcipriani: marking integration-agent-1080 offline for experimentation * 19:30 thcipriani: reconfiguring castor-save-workspace-cache with https://gerrit.wikimedia.org/r/1273935 * 17:47 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]]) * 16:49 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.32.10-do.1 to 1.32.13-do.2 ([[phab:T423726|T423726]]) === 2026-04-16 === * 20:49 dduvall: creating integration/zuul-jobs repo to serve as a mirror of opendev.org/zuul/zuul-jobs ([[phab:T406384|T406384]]) * 13:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272711 [[phab:T423568|T423568]] * 11:07 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud === 2026-04-15 === * 20:05 James_F: Zuul: Configure REL1_46 CI, for [[phab:T423257|T423257]] * 17:44 bd808: Unblock 176.0.0.0/13 * 17:39 bd808: Unblock 46.128.0.0/16 * 17:32 bd808: Unblock 176.86.0.0/16 * 16:39 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/127d783b2176ac60b646a5fa4f1b1a872ca66340 * 15:33 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/100 * 01:02 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/99 === 2026-04-14 === * 20:42 James_F: Docker: [composer-scratch] Upgrade composer to 2.9.7 and cascade * 16:35 bd808: Unblock 88.112.0.0/14 * 00:48 bd808: Unblock 24.6.0.0/16 * 00:42 bd808: Unblock 152.231.48.0/20 === 2026-04-13 === * 22:00 James_F: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches * 20:28 James_F: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used * 14:42 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep, for [[phab:T421011|T421011]] === 2026-04-12 === * 18:21 James_F: jforrester@contint1002:~$ sudo /usr/sbin/service zuul restart && tail -f -n100 /var/log/zuul/zuul.log # [[phab:T423027|T423027]] === 2026-04-10 === * 23:22 James_F: jforrester@contint1002:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/ReadingLists --change {{Gerrit|1269498}},2 # [[phab:T422976|T422976]] * 23:20 James_F: Zuul: [mediawiki/extensions/ReadingLists] Publish JS coverage, for [[phab:T422976|T422976]] * 23:13 James_F: Zuul: Migrate a few straggler Node 20 MediaWiki things to Node 24 * 23:01 James_F: Zuul: Move all MediaWiki things from mediawiki-node20 to mediawiki-node24 * 21:59 James_F: Docker: Bump Node base images to March releases and cascade; Upgrade Quibble images from Node 20 to Node 24 * 10:24 hashar: Updating all Quibble jobs to 1.17.1 * 10:22 hashar: Updated PostgreSQL jobs to Quibble 1.17.1 # [[phab:T422110|T422110]] * 10:22 hashar: Updated apitesting job to Quibble 1.17.1 # [[phab:T422843|T422843]] [[phab:T418743|T418743]] * 09:51 hashar: Tag Quibble 1.17.1 @ {{Gerrit|0a1ab3b7c3dfee36c9bc2e9b049957d94e190e85}} === 2026-04-09 === * 15:13 hashar: Rolling back Quibble jobs to 1.16.0 (api-testing stage fails due to missing npm install step` * 14:58 hashar: Upgrading Quibble jobs to 1.17.0 * 14:23 hashar: Tagged Quibble 1.17.0 @ {{Gerrit|864381c6b63bdbcd8c74a3162c406fffcaaf8694}} * 07:48 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 "Zuul: use standalone jobs for GrowthExperiments Cypress tests" {{!}} [[phab:T417412|T417412]] === 2026-04-08 === * 22:19 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1269068 * 22:01 bd808: Unblock 95.216.12.170/32 ([[phab:T422751|T422751]]) * 19:26 brennen: gitlab-webhooks: building & deploying https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/37 - hitting some build tooling stuff, trying a fix per instructions in the error log * 17:54 bd808: Unblock 167.56.0.0/13 ([[phab:T422721|T422721]]) * 06:31 hashar: Deleted integration-agent-castor05 Bullseye instance, replaced by integration-agent-castor06 which is on Bookworm # [[phab:T421114|T421114]] * 06:24 hashar: Deleted integration-agent-qemu-1003 Bullseye image, replaced by integration-agent-qemu-1004 which is on Bookworm # [[phab:T422488|T422488]] === 2026-04-07 === * 22:25 dduvall: adding new pipelinelib labels to ci nodes ([[phab:T422234|T422234]]) * 20:05 hashar: Triggered a build of https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/ * 17:06 dduvall: added `Docker` label to `contint` jenkins nodes ([[phab:T422507|T422507]]) * 17:05 dduvall: restored missing `pipelinelib` labels on `integration-agent-docker-` CI hosts ([[phab:T422507|T422507]]) * 16:53 bd808: Unblock 73.0.0.0/8 ([[phab:T422498|T422498]]) * 12:36 hashar: jjb: use $CASTOR_HOST for Quibble success cache. https://gerrit.wikimedia.org/r/1268545 {{!}} This causes the Quibble jobs to use a new instance for the success cache, which is empty # [[phab:T383243|T383243]] [[phab:T421114|T421114]] * 12:17 hashar: Migrated Castor from integration-castor05 to integration-castor06. Updated CASTOR_HOST in Jenkins and moved the Cinder volume to the new instance # [[phab:T421114|T421114]] * 11:14 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1090, 1091, 1092 and 1093 # [[phab:T421114|T421114]] * 10:09 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1083 to 1089 # [[phab:T421114|T421114]] * 07:23 hashar: CI Jenkins: removed `blubber` label from all agents after having moved PipelineLib to use the `Docker` label {{!}} [[phab:T422234|T422234]] === 2026-04-06 === * 16:01 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1268239 === 2026-04-03 === * 20:17 bd808: Unblock 2.54.0.0/16 ([[phab:T422238|T422238]]) * 17:25 bd808: Unblock 31.18.0.0/16 ([[phab:T422245|T422245]]) * 17:18 bd808: Unblock 2.54.128.0/19 ([[phab:T422238|T422238]]) * 16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1264649 "add Python 3.14 to pywikibot jobs and separate lint tests" {{!}} [[phab:T421723|T421723]] * 09:26 hashar: integration: nuked pywikibot/core pre-commit cache # [[phab:T422242|T422242]] * 09:15 hashar: Added Bookworm based Jenkins agents to the pool with label `Docker`. Hostnames are `integration-agent-docker-107*` # [[phab:T421114|T421114]] * 02:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1267398 === 2026-04-02 === * 16:50 thcipriani: restart jenkins * 15:15 bd808: Unblock 82.216.0.0/16 ([[phab:T421508|T421508]]) * 15:07 bd808: Unblock 95.90.0.0/15 ([[phab:T421485|T421485]]) * 11:19 James_F: Zuul: [oojs/ui] Drop ooui-ruby2.7-rake job, we're abandoning Ruby use there === 2026-04-01 === * 22:01 bd808: Unblock 109.144.0.0/12 ([[phab:T422019|T422019]]) * 20:16 bd808: Unblock 93.192.0.0/10 ([[phab:T421894|T421894]]) * 19:25 dancy: Updating buildkitd to v0.29.0 in gitlab-cloud-runners (prod) ([[phab:T415284|T415284]]) * 17:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/97 ([[phab:T420441|T420441]]) * 17:39 bd808: Unblock 94.134.0.0/15 ([[phab:T421866|T421866]]) * 16:31 dancy: Upgrade buildkit to 0.29.0 in staging gitlab-cloud-runners ([[phab:T415284|T415284]]) * 10:47 taavi: integration-castor05: free up a bit of disk space by deleting cache for AhoCorasick/ CLDRPluralRuleParser/ HtmlFormatter/ RelPath/ RunningStat/ IPSet/ === 2026-03-30 === * 22:01 bd808: Unblock 78.20.0.0/14 ([[phab:T421586|T421586]]) * 21:04 bd808: Unblock 95.88.0.0/15 ([[phab:T421774|T421774]]) * 20:49 bd808: Unblock 95.89.191.0/24 ([[phab:T421774|T421774]]) * 20:29 bd808: Unblock 73.162.0.0/16 ([[phab:T421549|T421549]]) * 13:10 hashar: gerrit: abandon mediawiki/core changes that are 2+years old and are attached to a task (`Bug: Txxxx`) * 11:37 hashar: Reloaded Zuul to to add 3 persons to the allow list * 10:43 James_F: Docker: Re-pushing to try to create quibble-coverage 1.16.0-s2 === 2026-03-27 === * 21:00 James_F: Docker: [quibble-bullseye] Drop Python 2 from images * 11:28 hashar: deployment-prep: removed block for `143.176.0.0/15` and blocked subblock `143.176.0.0/16` instead. This unblocks `143.177.0.0/16` # [[phab:T421420|T421420]] * 00:18 bd808: Unblock 95.90.238.0/23 ([[phab:T421447|T421447]]) === 2026-03-26 === * 21:25 bd808: Unblock 89.240.0.0/15 ([[phab:T421364|T421364]]) * 21:09 brennen: patchdemo: deploy to production for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312 === 2026-03-25 === * 20:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256318 [[phab:T421283|T421283]] * 15:46 dancy: Migrated gitlab-cloud-runners (prod) from nginx-ingress to traefik ([[phab:T420743|T420743]]) * 15:32 dancy: Migrated gitlab-cloud-runners (staging) from nginx-ingress to traefik ([[phab:T420743|T420743]]) * 10:01 hashar: Updating tox Jenkins jobs to add support for Python 3.14 {{!}} https://gerrit.wikimedia.org/r/1260632 {{!}} [[phab:T421209|T421209]] * 08:40 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20/ === 2026-03-24 === * 19:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255746 * 15:34 brennen: gitlab1004: manual test run of `configure-projects` with cleared issue allowlist ([[phab:T412882|T412882]]) * 15:26 bd808: Unblock 47.194.0.0/16 ([[phab:T421127|T421127]]) * 12:53 hashar: integration: deleted old Puppet 5 compiler agents from Jenkins ( pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud ) # [[phab:T367399|T367399]] * 07:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1259755 === 2026-03-23 === * 15:28 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 90272 === 2026-03-22 === * 14:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1258082 * 01:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256488 === 2026-03-21 === * 08:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256962 * 07:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256946 === 2026-03-20 === * 21:21 bd808: Unblock 103.159.218.0/24 ([[phab:T420530|T420530]]) * 14:59 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror, for [[phab:T399673|T399673]] === 2026-03-19 === * 16:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255777 * 16:01 Krinkle: Hoist l10n-bot rights from labs/tools parent to labs parent to reduce duplication in other labs/ repos * 15:50 Krinkle: Create labs/xtools repo (branch: main, parent: labs, owner: labs-xtools), ref [[phab:T402086|T402086]] === 2026-03-18 === * 21:11 dcausse: [[phab:T403775|T403775]]: reindexing all wikis to enable new sorting options * 21:08 dcausse: restarting opensearch on deployment-cirrussearch(12{{!}}13{{!}}14) instances to pickup new plugin versions * 14:56 James_F: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest * 14:52 James_F: Zuul: [GrowthExperiments] drop duplicate VisualEditor dep * 14:52 James_F: Zuul: [search/*] Add experimental Java 25 jobs === 2026-03-17 === * 22:50 James_F: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs * 21:27 James_F: Zuul: search: Update opensearch plugins for Java 11/17, for [[phab:T420407|T420407]] * 20:20 bd808: Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 ([[phab:T415021|T415021]]) * 16:43 James_F: Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps * 14:36 James_F: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep, for [[phab:T420015|T420015]] === 2026-03-13 === * 13:59 andrewbogott: deleting ptr record 117.0.16.172.in-addr.arpa. -- accidental duplicate for deployment-kafka-logging01.deployment-prep.eqiad1.wikimedia.cloud * 13:04 elukey: re-create kafka-logging-01 in deployment-prep on trixie and Kafka 3.7 (was running on buster) * 09:13 elukey: upgrade kafka-jumbo and kafka-main to Confluent 7.7 in deployment-prep (pre-requisite before being able to upgrade to Trixie) === 2026-03-12 === * 21:23 bd808: Hard reboot deployment-sessionstore06 ([[phab:T415021|T415021]]) * 01:14 James_F: Docker: [helm-linter] Bump for Envoy 1.35.9, for [[phab:T419637|T419637]] === 2026-03-11 === * 16:48 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/MetricsPlatform # [[phab:T417568|T417568]] * 16:47 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Archive, for [[phab:T416865|T416865]] * 11:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1250529 "inference-services: Split policy violation CI into separate model jobs." - [[phab:T418832|T418832]] === 2026-03-10 === * 17:39 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner production * 17:11 hashar: Updated MediaWiki coverage jobs so that they now keep "Generate a local configuration by running `composer phpunit:config`" message # [[phab:T419073|T419073]] * 16:41 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner staging * 08:21 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 === 2026-03-09 === * 21:53 bd808: Reboot deployment-shellbox01 on the off chance that is makes the new permissions error go away ([[phab:T419440|T419440]]) * 13:13 James_F: Zuul: [mediawiki/extensions/WikiShare] Mark as archived, for [[phab:T413589|T413589]] * 13:11 James_F: Zuul: [mediawiki/extensions/Memento] Mark as archived, for [[phab:T369991|T369991]] * 13:10 James_F: Zuul: [mediawiki/extensions/QuickGV] Mark as archived, for [[phab:T413348|T413348]] * 13:10 James_F: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived, for [[phab:T413588|T413588]] * 13:09 James_F: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived, for [[phab:T413587|T413587]] * 13:07 James_F: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived, for [[phab:T413586|T413586]] * 13:06 James_F: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived, for [[phab:T413585|T413585]] * 13:04 James_F: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived, for [[phab:T418008|T418008]] * 12:50 James_F: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency * 12:50 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen * 10:40 hashar: gerrit: mediawiki/vendor: converted `es6` and `es710` branches to tags # [[phab:T417804|T417804]] * 09:24 hashar: Updating Quibble jobs to 1.16.0 {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1248880 {{!}} [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]] * 09:15 hashar: updating all CI Jenkins jobs using `./jjb-update` === 2026-03-06 === * 19:46 James_F: Zuul: [mediawiki/services/geoshapes] Mark as archived, for [[phab:T418372|T418372]] * 16:37 hashar: Building Docker images for Quibble 1.16.0 * 16:31 hashar: Tag Quibble 1.16.0 @ {{Gerrit|0b9db5fe3cabb2cec0b5d44e128bafa917b3b895}} # [[phab:T417399|T417399]] [[phab:T417409|T417409]] [[phab:T418461|T418461]] * 12:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248411 "jjb, Zuul: vary Wikibase Selenium for release branches" {{!}} [[phab:T418797|T418797]] * 12:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248409/ "jjb, Zuul: rename wikibase-selenium job for clarity" {{!}} [[phab:T418797|T418797]] === 2026-03-05 === * 14:41 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add TestKitchen as a dependency for [[phab:T418053|T418053]] * 08:01 hashar: Reloaded Zuul to rename wikibase-client / wikibase-repo jobs {{!}} https://gerrit.wikimedia.org/r/1238317 * 00:04 James_F: Docker: [quibble-coverage] Use local PHPUnit config, for [[phab:T345481|T345481]] === 2026-03-04 === * 21:16 James_F: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]] * 21:10 James_F: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch, for [[phab:T411814|T411814]] * 19:48 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 ([[phab:T419004|T419004]]) * 18:50 James_F: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency", for [[phab:T419043|T419043]] * 16:23 James_F: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting * 15:37 James_F: Docker: [rake-ruby2.7] Add libffi-dev too, for [[phab:T418463|T418463]] * 13:59 James_F: Docker: [rake-ruby2.7] Add ruby-ffi for [[phab:T418463|T418463]] * 13:54 hashar: SIGKILL Zuul cause it can't gracefully stop most probably due to being locked attempting to report back to Gerrit # [[phab:T419009|T419009]] * 13:49 hashar: Stopping Zuul # [[phab:T419009|T419009]] * 13:41 hashar: Took a Zuul stack dump on contint1002.wikimedia.org using SIGUSR1 # [[phab:T419009|T419009]] === 2026-03-03 === * 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Drop MetricsPlatform phan dep * 23:52 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Drop MetricsPlatform phan dep === 2026-03-02 === * 22:13 James_F: Zuul: Enforce PHP 8.4 in MW extensions and skins for development branch, for [[phab:T386108|T386108]] * 14:05 James_F: Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency, for [[phab:T415451|T415451]] * 13:48 James_F: Zuul: […/WikimediaEvents] Drop LoginNotify dependency, now unused, for [[phab:T404334|T404334]] * 10:16 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/15.8.2/ # [[phab:T418718|T418718]] === 2026-02-28 === * 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/skins` # [[phab:T418675|T418675]] * 21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/extensions` # [[phab:T418675|T418675]] === 2026-02-27 === * 15:53 dancy: Updating gitlab-cloud-runners (staging and prod) to gitlab-runner 18.9.0. === 2026-02-26 === * 20:16 James_F: Zuul: Provide a custom, high-priority pipeline just for puppet compiler [[phab:T414621|T414621]] * 19:32 James_F: Docker: Bump all the PHPs. * 13:40 hashar: Deployed Jenkins job https://integration.wikimedia.org/ci/job/wikibase-selenium/ # [[phab:T287582|T287582]] * 00:13 dduvall: forcing replacement of buildkitd helm release in gitlab-cloud-runner prod cluster due to dependency on removed k8s secret ([[phab:T416260|T416260]]) === 2026-02-25 === * 23:50 dduvall: deploying https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 to gitlab-cloud-runner production cluster ([[phab:T416260|T416260]]) * 14:07 James_F: Zuul: [mediawiki/extensions/CommunityRequests] Add TemplateData dependency, for [[phab:T401638|T401638]] * 00:08 jeena: no-op testing updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95 === 2026-02-24 === * 15:55 brennen: devtools: test deploy phab/phorge to test instance ([[phab:T418256|T418256]]) === 2026-02-23 === * 23:07 jeena: Updated development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 * 22:43 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 * 22:12 bd808: Unblock 191.80.192.0/18 ([[phab:T418132|T418132]]) * 20:26 hashar: Deleted "replication-upstream" Grafana dashboard in favor of a copy/new "replication" one. https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream , replaced it by https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication * 16:29 James_F: Zuul: [3d2png] Add basic Node CI at version 20 === 2026-02-20 === * 21:47 bd808: Unblock 168.184.84.0/24 ([[phab:T418020|T418020]]) * 17:13 bd808: Unblock 122.187.64.0/18 ([[phab:T417964|T417964]]) * 14:35 James_F: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section === 2026-02-19 === * 18:34 bd808: Unblock 181.98.0.0/16 ([[phab:T417890|T417890]]) * 17:21 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add AbuseFilter as a dependency, for [[phab:T417799|T417799]] * 13:22 hashar: Reloaded Zuul to archive the Cergen repository {{!}} https://gerrit.wikimedia.org/r/c/integration/config/+/1240688 {{!}} [[phab:T417887|T417887]] === 2026-02-18 === * 20:17 jeena: Updating development images on contint primary for [[phab:T415922|T415922]] * 19:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240360 * 18:40 bd808: Unblock 46.59.0.0/17 ([[phab:T417747|T417747]]) * 17:05 hashar: Regenerating Jenkins jobs with JJB based on https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/ * 17:04 hashar: Added EXT_DEPENDENCIES to Quibble Jenkins jobs parameters so we can manually trigger them from the Web UI using a different set of deps # https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/ * 16:30 hashar: Triggered https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ with empty Zuul parameters introduced by https://gerrit.wikimedia.org/r/1240333 {{!}} https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4893/console * 15:43 James_F: Zuul: [mediawiki/extensions/ReadingLists] Add EventBus dependency for [[phab:T417706|T417706]] * 12:15 hashar: zuul-1001.zuul3.eqiad1.wikimedia.cloud: added keepalive=20 to the scheduler Gerrit driver and restarted scheduler container # [[phab:T417497|T417497]] * 06:58 jeena: Updating development images on contint primary for [[phab:T415922|T415922]] === 2026-02-17 === * 23:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240081 * 23:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240078 * 15:58 brennen: deployed latest phab/phorge wmf/stable to devtools test instance ([[phab:T417657|T417657]]) * 09:01 hashar: Reloaded Zuul to enable php 8.5 testing on utfnormal, php-session-serializer, wikipeg, mediawiki/libs/Dodo, mediawiki/libs/UUID, testing-access-wrapper and translatewiki # [[phab:T406326|T406326]] === 2026-02-16 === * 15:27 hashar: Manually cleaned some old workspaces on integration-agent-docker-1042 === 2026-02-12 === * 20:07 James_F: Zuul: Enable PHP 8.5 jobs for most MW libraries, for [[phab:T406326|T406326]] * 19:33 James_F: Docker: [php83] Re-build with upstream's new 8.3.30 release and cascade * 19:31 James_F: Zuul: Add PHP 8.5 CI job to various things noted as blocked by Phan, for [[phab:T410941|T410941]], [[phab:T406326|T406326]] * 16:35 Krinkle: Disable publishing noise on tasks from repos Bcp47, clover-diff, ScopedCallback, and IDLeDOM. Ref [[phab:T143162|T143162]] * 15:53 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/87 * 11:21 James_F: Zuul: [mediawiki/libs/shellbox] Add direct Phan job, for [[phab:T416064|T416064]] === 2026-02-10 === * 20:16 dancy: Rebooted k3s.catalyst-dev (it was unresponsive, but the reboot hasn't helped) === 2026-02-09 === * 21:58 James_F: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job, for [[phab:T410941|T410941]] * 19:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1238006 [[phab:T415680|T415680]] * 11:51 James_F: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency, for [[phab:T414435|T414435]] === 2026-02-05 === * 17:58 James_F: Zuul: […/WikimediaCustomizations] Add six new dependencies for [[phab:T404334|T404334]] * 15:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1237254 * 15:18 James_F: Zuul: […/OATHAuth] Add dependency and phan dependency on CentralAuth === 2026-02-04 === * 12:54 James_F: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency * 10:03 hashar: Restarted Jenkins on releases2003.codfw.wmnet === 2026-02-02 === * 21:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1234926 "re-enable master jobs for some BlueSpice repos - [[phab:T403196|T403196]]" * 21:05 bd808: Unblock 85.146.0.0/17 ([[phab:T416079|T416079]]) * 19:47 James_F: Zuul: […/WikimediaCustomizations] Add cldr phan dependency, for [[phab:T404334|T404334]] * 17:33 bd808: Unblock 188.188.0.0/15 ([[phab:T416095|T416095]]) * 17:26 bd808: Unblock 85.94.84.0/22 ([[phab:T416105|T416105]]) * 17:09 bd808: Unblock 94.234.0.0/16 ([[phab:T416165|T416165]]) * 16:51 dancy: Update gitlab-runners to alpine-v18.6.6 ([[phab:T415214|T415214]]) * 16:27 bd808: Unblock 47.231.208.0/21 ([[phab:T416010|T416010]]) * 11:39 James_F: Zuul: […/WikimediaCustomizations] Add five new phan dependencies, for [[phab:T404334|T404334]] * 09:45 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 58532, 58557 === 2026-01-31 === * 21:49 James_F: Deleted Jenkins's job entry for castor-save-workspace-cache {{Gerrit|6193776}} and this seems to have unstuck things for [[phab:T416078|T416078]]? * 21:45 James_F: Running `sudo systemctl restart jenkins` on contint for [[phab:T416078|T416078]] * 21:44 James_F: Fighting [[phab:T416078|T416078]], took integration-castor-5 offline, disconnected, sshed in to kill threads, then reconnected; no change in aspect. * 19:03 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1235380 === 2026-01-28 === * 21:26 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn # [[phab:T415832|T415832]] * 21:11 bd808: Unblock 181.160.0.0/15 & 186.40.128.0/17 ([[phab:T415820|T415820]]) * 17:01 bd808: Unblock 102.182.0.0/16 ([[phab:T415782|T415782]]) === 2026-01-27 === * 16:45 James_F: Zuul: Switch skin-quibble template with identical extension-quibble, for [[phab:T402398|T402398]] * 16:18 James_F: Zuul: [ArticleGuidance] mention it will be in production * 15:55 James_F: Docker: [quibble-bullseye] Update to Quibble 1.15.0 * 15:12 James_F: Docker: [quibble-coverage] Pass PHPUnit config location explicitly, for [[phab:T395470|T395470]] * 09:18 hashar: integration: on integration-castor05, deleted caches for old MediaWiki branches * 09:15 hashar: integration: on pkgbuilder instances, removed Buster cow images, aptcache and hooks. `sudo cumin --force -p 0 'name:pkgbuilder' 'rm -fR /srv/pbuilder/<nowiki>{</nowiki>base-buster-amd64.cow,hooks/buster,aptcache/buster-amd64<nowiki>}</nowiki>'` # [[phab:T397209|T397209]] * 09:14 hashar: integration: cleaned up old workspaces under /srv/jenkins/workspace === 2026-01-26 === * 23:27 bd808: Unblock 66.130.0.0/15 ([[phab:T415596|T415596]]) * 22:52 bd808: Unblock 45.16.0.0/12 ([[phab:T415467|T415467]]) * 14:46 hashar: gerrit: changed `operations/software/permissions` project type from `CODE` to `PERMISSIONS` by pointing `HEAD` to `refs/meta/config` === 2026-01-22 === * 17:36 James_F: Docker: [quibble-coverage] Stop using legacy PHPUnit entrypoint ([[phab:T395470|T395470]]) & Stop excluding Dump/ParserFuzz/Stub groups ([[phab:T415230|T415230]]) * 15:11 James_F: Zuul: [mediawiki/extensions/Math] Add a standalone job, for [[phab:T415230|T415230]] === 2026-01-20 === * 20:38 bd808: Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229186 ([[phab:T415113|T415113]]) * 19:05 bd808: Rebooted deployment-cache-text08 to see if the mystery haproxy startup failure would go away ([[phab:T415100|T415100]]) * 18:50 bd808: Unblock 152.7.0.0/16 ([[phab:T415100|T415100]]) === 2026-01-17 === * 23:32 ori: beta-scap with `php_l10n: true` completed successfully: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/241466/console. PHP l10n files generated. Reverted local change to scap.cfg. * 23:26 ori: Temporarily set `php_l10n: true` on deployment-deploy04:/etc/scap.cfg to see if next scap succeeds. === 2026-01-16 === * 16:33 dancy: Deleting deployment-mx03.deployment-prep ([[phab:T412975|T412975]]) === 2026-01-15 === * 14:50 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/ArticleSummaries/ # [[phab:T413232|T413232]] === 2026-01-14 === * 17:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226907 * 16:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226893 * 15:57 bd808: Unblock 190.60.63.0/24 ([[phab:T414541|T414541]]) === 2026-01-13 === * 15:04 James_F: Zuul: Make quibble-for-mediawiki-core-vendor-mysql-php84 voting, for [[phab:T386108|T386108]] === 2026-01-12 === * 21:33 zabe: zabe@deployment-mwmaint03:~$ foreachwiki migrateLinksTable.php --table imagelinks # [[phab:T413668|T413668]] * 21:06 bd808: Unblock 66.81.168.0/21 ([[phab:T414303|T414303]]) * 17:42 dancy: Turned off instance deployment-prep.deployment-mx03 * 11:44 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 46331, 46344 === 2026-01-10 === * 21:48 taavi: reload zuul for https://gerrit.wikimedia.org/r/1224782 * 00:25 bd808: Unblock 91.160.0.0/12 ([[phab:T414190|T414190]]) === 2026-01-09 === * 17:33 thcipriani: re-enabling beta update jobs after test bad extension-list [[phab:T411516|T411516]] * 17:09 thcipriani: disabling beta update jobs to test bad extension-list [[phab:T411516|T411516]]) === 2026-01-08 === * 21:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224815 [[phab:T414136|T414136]] * 18:24 bd808: Unblock 89.80.0.0/12 ([[phab:T414113|T414113]]) * 15:55 dancy: Upgrading gitlab-runner to v18.5.0 on gitlab-cloud-runners. ([[phab:T414053|T414053]]) === 2026-01-07 === * 23:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1082574 https://gerrit.wikimedia.org/r/1224157 https://gerrit.wikimedia.org/r/1224159 * 23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896311 [[phab:T27482|T27482]] * 23:06 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224218 * 17:34 James_F: Zuul: Add new extensions: IssueTrackerLinks, PreviewLinks, and WikiRAG * 17:34 James_F: Zuul: [labs/tools/heritage] Point to the task to drop 8.1 testing * 15:09 James_F: Zuul: [labs/tools/heritage] Add testing in PHP 8.2+, not just PHP 8.1 * 15:03 James_F: Zuul: Even for extension-broken, don't offer PHP 8.1 testing * 15:02 James_F: Zuul: Move quibble experimental sqlite/postgres tests to PHP 8.3 === 2026-01-06 === * 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223690 [[phab:T411814|T411814]] * 16:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223189 [[phab:T411814|T411814]] * 00:30 bd808: Unblock 85.134.128.0/17 ([[phab:T413755|T413755]]) * 00:02 bd808: Unblock 89.166.128.0/17 ([[phab:T413702|T413702]]) === 2026-01-05 === * 23:57 bd808: Unblock 185.233.104.0/22 ([[phab:T413472|T413472]]) * 23:51 bd808: Unblock 45.62.112.0/21 ([[phab:T413079|T413079]]) * 23:44 bd808: Unblock 85.134.200.0/21 ([[phab:T413067|T413067]]) * 19:03 dancy: Updated buildkitd to v0.26.3 in gitlab-cloud-runners * 14:27 taavi: reload zuul for {{Gerrit|1223191}} * 13:57 James_F: Zuul: [mediawiki/php/wmerrors] Enable PHP 8.5 testing, for [[phab:T410921|T410921]] === 2026-01-03 === * 17:59 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222709 https://gerrit.wikimedia.org/r/1220388 https://gerrit.wikimedia.org/r/1219140 === 2026-01-02 === * 17:10 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1222597 === 2026-01-01 === * 02:34 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1221644 <noinclude>'''Server Admin Log''' logged from {{IRC|wikimedia-releng}} for [[Nova Resource:Deployment-prep|Beta Cluster]], [[mw:Continuous integration|Continuous integration]] and various other Release Engineering projects.</noinclude> {{SAL-archives/Release Engineering}} <noinclude>[[Category:SAL]]</noinclude> m699lzwnfr87q5cr89pb0zxsts09dq4 0 91251 2452858 2430539 2026-08-31T10:17:02Z MMuhlenhoff (WMF) 5268 /* Service */ Document the new LVS setup 2452858 wikitext text/x-wiki {{Warn|content=Please, never use this functionality to reach out to endpoints belonging to Wikimedia. You should be able to utilize the service mesh/services proxy to reach most endpoints internally in a safer, more reliable and more performing way. If you can't find out how, please reach out to SRE}} '''url-downloader''' is a [[Squid]] non-caching proxy for HTTP requests. It was originally created for MediaWiki's upload-by-URL functionality, these days it is being used by any service that wants to access resources outside the boundaries of the wikimedia production realm. It is not a highly available service and as such no assumptions should be made in code regarding availability. == Service == We have a total of four URL downloader VMs running in eqiad and codfw (urldownloader[12]00[56].wikimedia.org), which are load-balanced behind LVS.The endpoints to access them are: * urldownloader.discovery.wmnet (discovery DNS record) * urldownloader.svc.eqiad.wmnet (site-specific endpoint in eqiad) * urldownloader.svc.codfw.wmnet (site-specific endpoint in codfw) In the past only of the URL downloader nodes was accessed via CNAME DNS records; <code>url-downloader.$DC.wikimedia.org</code> and <code>url-downloader.wikimedia.org</code>. These legacy endpoints still point to individual nodes, but should no longer be used as they will eventually be removed entirely. For technical details please also refer to [https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/url_downloader.pp url_downloader] the Puppet profile.'' == Administration == === Adding/replacing URL downloader nodes === Any server in the production networks can directly access a new URL downloader proxy after applying the Puppet role. No changes in homer/on the routers are necessary. Access to a specific node can be tested with e.g. <pre> curl -x http://urldownloader100X.wikimedia.org:8080 $URL > download </pre> In addition the new node (or the node to be removed) need to be added/removed from the egress rules for Kubernetes. An example commit can be found at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1303341. Following the merge this need to be deployed via the admin_ng section of the deployment charts. This enables access to all our Kubernetes clusters, not just wikikube. <pre> sudo -i cd /srv/deploymenmt/deployment-charts/helmfile.d/admin_ng helmfile -e [staging-eqiad|staging-codfw|eqiad|codfw] -i [diff|apply] </pre> The change need to be applied to all ''external-services'' and ''calico''. == See also == * [[HTTP proxy]] (another set of squid proxies for slightly different use cases) * [[phab:T254011|T254011: Why do we have 2 sets of squid proxies?]] {{lowercase}} [[Category:Services]] dbkfsb64xkr2e5i9etr5ifm6y7zf7lm 2452866 2452858 2026-08-31T10:24:53Z MMuhlenhoff (WMF) 5268 Add a brief troubleshooting section with the basics 2452866 wikitext text/x-wiki {{Warn|content=Please, never use this functionality to reach out to endpoints belonging to Wikimedia. You should be able to utilize the service mesh/services proxy to reach most endpoints internally in a safer, more reliable and more performing way. If you can't find out how, please reach out to SRE}} '''url-downloader''' is a [[Squid]] non-caching proxy for HTTP requests. It was originally created for MediaWiki's upload-by-URL functionality, these days it is being used by any service that wants to access resources outside the boundaries of the wikimedia production realm. It is not a highly available service and as such no assumptions should be made in code regarding availability. == Service == We have a total of four URL downloader VMs running in eqiad and codfw (urldownloader[12]00[56].wikimedia.org), which are load-balanced behind LVS.The endpoints to access them are: * urldownloader.discovery.wmnet (discovery DNS record) * urldownloader.svc.eqiad.wmnet (site-specific endpoint in eqiad) * urldownloader.svc.codfw.wmnet (site-specific endpoint in codfw) In the past only of the URL downloader nodes was accessed via CNAME DNS records; <code>url-downloader.$DC.wikimedia.org</code> and <code>url-downloader.wikimedia.org</code>. These legacy endpoints still point to individual nodes, but should no longer be used as they will eventually be removed entirely. For technical details please also refer to [https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/url_downloader.pp url_downloader] the Puppet profile.'' == Troubleshooting == The service runs on port 8080 and is provided by a backport of Squid 7.6 to trixie. Cache access is logged to /var/log/squid/access.log and potential errors would end up in the journald of squid.service or /var/log/squid/cache.log. The URL downloader nodes are behind LVS, so if there's any load issues also make sure to validate if one of the nodes listed in conf-tool might be depooled (accidentally or intentionally). == Administration == === Adding/replacing URL downloader nodes === Any server in the production networks can directly access a new URL downloader proxy after applying the Puppet role. No changes in homer/on the routers are necessary. Access to a specific node can be tested with e.g. <pre> curl -x http://urldownloader100X.wikimedia.org:8080 $URL > download </pre> In addition the new node (or the node to be removed) need to be added/removed from the egress rules for Kubernetes. An example commit can be found at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1303341. Following the merge this need to be deployed via the admin_ng section of the deployment charts. This enables access to all our Kubernetes clusters, not just wikikube. <pre> sudo -i cd /srv/deploymenmt/deployment-charts/helmfile.d/admin_ng helmfile -e [staging-eqiad|staging-codfw|eqiad|codfw] -i [diff|apply] </pre> The change need to be applied to all ''external-services'' and ''calico''. == See also == * [[HTTP proxy]] (another set of squid proxies for slightly different use cases) * [[phab:T254011|T254011: Why do we have 2 sets of squid proxies?]] {{lowercase}} [[Category:Services]] g42khu70h8ejl3hbpggfs4i8dtwsx16 0 442914 2452780 2440755 2026-08-30T22:00:59Z Akd.2303 58755 /* sincronizar */ 2452780 wikitext text/x-wiki The Dashiki dashboarding system depends on pages on meta.wikimedia.org to configure dashboards and related metadata, such as annotations. Here we describe these configuration files and the schemas they use. Currently, these schemas are not enforced when you save an article, so users are responsible for making sure the configuration works. == Annotations == Articles that specify annotations to use on dashboards need to have a path such as <pre>Config:Dashiki:Annotations/<<project name>>/<<metric name>></pre> Where <<project name>> is something like Wikistats and <<metric name>> is something like editors. So, a concrete example is m:Config:Dashiki:Annotations/Wikistats/totalPageViews. Annotations can be defined here in the following format: description: -- text describing this set of annotations documentation: -- link to the metric's research page or other context annotations: -- a list of annotations using one of these patterns: - date: title: note: - date: title: note: relevantBreakdowns: -- list of splits to limit annotations to - dimension: -- the split (eg. editor_type) or "total" for no breakdown values: -- list of breakdown values to limit to, eg.: - group-bot* <nowiki><ref>عنصر قائمة منقطة</ref></nowiki> - anonymous Take a look at the examples or ask questions on the talk page here. The slightly complicated thing is that if you choose to use relevantBreakdowns, you need to include the "total" dimension in the list, with values: [ "total" ] if you want to also show the breakdown on the main, un-split graph. == Dashboards == Articles that specify dashboard configurations need to have a path such as <pre> Config:Dashiki:<<fernandoesteban2408@gmail.com>> </pre> The format for the configuration depends on what type of layout your dashboard is using. These are detailed below: === Tabs Layout === === Compare Layout === === Metric by Project Layout === __NOTOC____DISAMBIG__ __NOINDEX__ __NONEWSECTIONLINK__ 05cxp45b5en1yxcv1reolomg0w6i2rq 0 449160 2452781 2452769 2026-08-31T00:02:17Z Dexbot 30554 Bot: Updating the report 2452781 wikitext text/x-wiki {{/Header}} == Today (2026-08-31) == == Yesterday (2026-08-30) == == Last seven days == [[Category:MariaDB]] 7o7kukeiy6tfjptzy93iv1sv0e7j9pg 0 453463 2452871 2415324 2026-08-31T10:33:37Z CGoubert-WMF 32775 /* Datastores */ Add redis-lock 2452871 wikitext text/x-wiki There are times when we need to reboot our fleet. Here are some notes to help us go through it like == Datastores == === [[Memcached for MediaWiki | Memcache]] cluster === Servers of this cluster need no depooling of any sort, and we have a pool of servers to pick up the traffic of any unavailable server. The cached data in the memecache cluster are crucial for our latency, so it is highly recommended to '''reboot 1 server at a time per DC''', with a sleep time of 15'-20' minimum between reboots, allowing the server to warm up. * https://grafana-rw.wikimedia.org/d/000000316/memcache?orgId=1 ==== Cookbook: <code>sre.memcached.roll-reboot-restart</code> ==== Use the [[Spicerack/Cookbooks|cookbook]] <code>sre.memcached.roll-reboot-restart</code> to perform rolling reboots or daemon restarts on memcached hosts. The cookbook automatically: * Verifies the gutter pool is healthy before touching the main pool (as failover capacity) * Confirms memcached is active on each host after the operation ===== Options ===== Standard cookbook options are used, with the addition of <code>--min-uptime</code>, so to provide some easy way to resume operations. * '''Currently the <code> --query </code> is broken, please only use aliases''' * <code>--min-uptime</code>: Only include hosts whose memcached service has been running at least this long (e.g. <code>7d</code>, <code>24h</code>, <code>604800</code>) ===== Examples ===== Rolling reboot of eqiad, one host at a time with 15-minute sleep (recommended defaults): <pre> cookbook sre.memcached.roll-reboot-restart --reason "Debian reboots" \ --alias memcached-eqiad --batchsize 1 --grace-sleep 900 reboot </pre> Restart only hosts whose memcached has been running for at least 10 days (e.g. to resume interrupted rolling restarts): <pre> cookbook sre.memcached.roll-reboot-restart --reason "Resume reloads" \ --alias memcached-eqiad --min-uptime 10d --batchsize 2 restart_daemons </pre> === [[Redis]] lock === Redis-lock hosts can be rebooted one after the other, MediaWiki can handle loss of one server at a time. There is no replication or complex mechanisms, just check the service is back up. Use the <code>sre.hosts.reboot</code> cookbook. === [[Redis]] misc === Redis hosts can be rebooted one after the other, taking care of waiting for replication to be back up after the reboot. Dashboards: [[https://grafana.wikimedia.org/d/000000174/redis?from=now-3h&orgId=1&timezone=utc&to=now&var-instance=$__all&var-job=redis_misc&var-site=$__all redis-misc on Grafana]] '''Please do not restart these servers during deployments because docker-registry uses redis''' ==== Service Upgrade ==== '''Upgrading redis-server''' A simple <code>apt-get install -y redis-server</code> is enough, but it will '''not''' trigger a restart, as we are using one systemd service per port, thus you will see the following message redis-server.service is a disabled or a static unit not running, not starting it.` '''Restarting instances''' Always restart '''replicas first''', then '''primaries second'''. Restart all instances one by one with a 5-second delay: <syntaxhighlight lang="bash"> rdb1001:# for port in 6378 6379 6380 6381 6382; do echo "Restarting $port..."; systemctl restart redis-instance-tcp_$port; sleep 5; done </syntaxhighlight> ==== Server reboot ==== Run the <code>sre.hosts.reboot</code> cookbook, taking care to reboot the replica first, and to wait for the replication status to be ok before moving to the primary. ==== Common checks ==== '''Checking replication status''' To check the replication status of all instances: <syntaxhighlight lang="bash"> rdb1001:# for port in 6378 6379 6380 6381 6382; do echo "--- $port ---"; redis-cli -p $port -a $(grep requirepass /etc/redis/tcp_$port.conf | awk '{print $2}') --no-auth-warning info replication | grep -E "master_link_status|master_last_io_seconds_ago|master_sync_in_progress"; echo "-----"; done </syntaxhighlight> Example output: <syntaxhighlight lang="text"> --- 6379 --- master_link_status:down master_last_io_seconds_ago:-1 master_sync_in_progress:1 ----- --- 6380 --- master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 ----- </syntaxhighlight> A healthy, caught-up replica should show: * <code>master_link_status:up</code> * <code>master_last_io_seconds_ago</code>: expect a low value (0-10) * <code>master_sync_in_progress:0</code> '''Full resync after restart''' After a restart, the replica will perform a '''full resync''' since it has no cached master state: <syntaxhighlight lang="text"> Master replied to PING, replication can continue... Partial resynchronization not possible (no cached master) Full resync from master: 7fe8b694f6afc193b57187a779dfd037c9884ee3:0 </syntaxhighlight> During this process, <code>master_last_io_seconds_ago:-1</code> indicates the replica has not yet connected or is still performing a full sync. '''Note:''' In the case of a '''primary''' redis server restart with a large dataset, it may take a while for replicas to reconnect, as the primary will need to load the full dataset into memory first. ==== docker-registry ==== <code>docker-registry</code> doesn't use HA for redis, so the reboot of its redis nodes will cause a short unavailability. '''Avoid restarting these nodes during deployments''', as the unavailability of the docker-registry redis could cause deployment failures. ---- === Etcd === == Kubernetes == === Production === ==== Workers ==== <code> sudo cookbook -d sre.k8s.reboot-nodes --batchsize 15 --k8s-cluster wikikube-codfw --reason "Reason" --alias wikikube-worker-codfw --minimal-cordon </code> <code> sudo cookbook -d sre.k8s.reboot-nodes --batchsize 15 --k8s-cluster wikikube-eqiad --reason "Reason" --alias wikikube-worker-eqiad --minimal-cordon </code> ==== Control plane ==== <code> sudo cookbook -d sre.k8s.reboot-nodes --batchsize 1 --k8s-cluster wikikube-eqiad --reason "Reason" --alias wikikube-master-eqiad --minimal-cordon </code> ==== Dragonfly supernodes==== Lock scap to avoid overloading the registry servers with a big deployment <code>deploy1003:~$ scap lock --all "Dragonfly supernodes reboot"</code> Then reboot using the <code>sre.hosts.reboot-single</code> cookbook === Staging === == Poolcounter == Each poolcounter server should be removed from <code>mediawiki-config/wmf-config/ProductionServices.php</code> with a gerrit patch before reboot, then that patch should be deployed to production using <code>scap backport</code>. After the reboot, add the server back and remove the next server in the same patch, <code>scap backport</code> it, and repeat until all poolcounter servers are rebooted and back in <code>mediawiki-config/wmf-config/ProductionServices.php</code> [[Thumbor]] uses poolcounter as well, but will fail open if its poolcounter server is down. You can forgo swapping the servers out in thumbor's <code>deployment-charts/helmfile.d/services/thumbor/values-{eqiad,codfw}.yaml</code> files if the interruption is short. == Chartmuseum == Chartmuseum is addressed through <code>helm-charts.discovery.wmnet</code>. It is backed by one VM in each datacentre. <syntaxhighlight lang=shell-session> # Depool codfw sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=codfw' set/pooled=false # Reboot codfw sudo cookbook sre.hosts.reboot-single -r "May 2025 Reboots" chartmuseum2001.codfw.wmnet # Repool codfw sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=codfw' set/pooled=true # Rinse and repeat for eqiad sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=eqiad' set/pooled=false sudo cookbook sre.hosts.reboot-single -r "May 2025 Reboots" chartmuseum1001.eqiad.wmnet sudo confctl --object-type discovery select 'dnsdisc=helm-charts.*,name=eqiad' set/pooled=true </syntaxhighlight> [[Category:SRE Service Operations]] 1c1lma0v7o21tmjiebnp8jiwmziete1 116 453906 2452872 2452080 2026-08-31T10:51:28Z Gitlabaccountapprovalbot 37332 @hgzh3 was approved. 2452872 wikitext text/x-wiki <noinclude>'''Audit log of approvals''' made by [[gitlab:gitlabaccountapprovalbot|@gitlabaccountapprovalbot]]. __NOTOC__</noinclude> === 2026-08-31 === * 10:51 [[gitlab:hgzh3|@hgzh3]] was approved. === 2026-08-27 === * 02:18 [[gitlab:madmax|@madmax]] was approved. === 2026-08-26 === * 21:42 [[gitlab:mmns21|@mmns21]] was approved. === 2026-08-25 === * 17:27 [[gitlab:skim|@skim]] was approved. * 14:57 [[gitlab:wilfredor2|@wilfredor2]] was approved. * 08:03 [[gitlab:noahmvf|@noahmvf]] was approved. === 2026-08-23 === * 05:30 "amidewicki" was rejected (pending since 2026-05-24T05:29:29.216Z). * 01:54 "westcodragen07" was rejected (pending since 2026-05-24T01:53:53.396Z). === 2026-08-22 === * 15:21 [[gitlab:laraibhasan|@laraibhasan]] was approved. === 2026-08-20 === * 17:18 [[gitlab:jatwa|@jatwa]] was approved. === 2026-08-18 === * 11:33 "wladek92" was rejected (pending since 2026-05-19T11:30:32.679Z). === 2026-08-17 === * 18:39 [[gitlab:erlanger16e|@erlanger16e]] was approved. * 18:33 "ashishkaranam" was rejected (pending since 2026-05-18T18:32:18.813Z). * 12:48 [[gitlab:lfs|@lfs]] was approved. === 2026-08-16 === * 15:42 [[gitlab:pcendrer|@pcendrer]] was approved. === 2026-08-15 === * 17:33 [[gitlab:flammablepizza|@flammablepizza]] was approved. === 2026-08-14 === * 13:00 [[gitlab:pacmand|@pacmand]] was approved. === 2026-08-12 === * 13:21 [[gitlab:doublechek|@doublechek]] was approved. === 2026-08-11 === * 14:15 [[gitlab:mmilenkovicwmf|@mmilenkovicwmf]] was approved. === 2026-08-10 === * 17:18 [[gitlab:mustafe|@mustafe]] was approved. * 10:33 "c8263a20" was rejected (pending since 2026-05-11T10:32:40.353Z). === 2026-08-09 === * 21:18 [[gitlab:iamnetx|@iamnetx]] was approved. * 20:09 "yirba" was rejected (pending since 2026-05-10T20:07:07.738Z). * 06:42 "marsam2489" was rejected (pending since 2026-05-10T06:40:46.276Z). === 2026-08-08 === * 08:15 [[gitlab:taiwaniajusto|@taiwaniajusto]] was approved. === 2026-08-07 === * 18:21 [[gitlab:gturkington|@gturkington]] was approved. * 06:36 "brianbybyby" was rejected (pending since 2026-05-08T06:36:08.459Z). === 2026-07-30 === * 20:42 "horaciocolbert" was rejected (pending since 2026-04-30T20:41:40.421Z). === 2026-07-29 === * 10:03 [[gitlab:piastu|@piastu]] was approved. * 06:33 "rafiul1" was rejected (pending since 2026-04-29T06:33:02.573Z). === 2026-07-28 === * 16:51 [[gitlab:for-each-next|@for-each-next]] was approved. * 09:21 [[gitlab:gka|@gka]] was approved. === 2026-07-27 === * 08:12 [[gitlab:cambob|@cambob]] was approved. === 2026-07-26 === * 15:39 "demansanaagmailcom" was rejected (pending since 2026-04-26T15:39:03.297Z). === 2026-07-24 === * 10:54 [[gitlab:ysogo|@ysogo]] was approved. * 09:39 [[gitlab:pankaj199|@pankaj199]] was approved. === 2026-07-23 === * 12:30 [[gitlab:fermiboson|@fermiboson]] was approved. * 09:21 [[gitlab:slashme|@slashme]] was approved. === 2026-07-22 === * 15:54 [[gitlab:lmedley|@lmedley]] was approved. * 13:24 "praveen5638" was rejected (pending since 2026-04-22T13:21:23.368Z). * 12:48 [[gitlab:cyberpower678|@cyberpower678]] was approved. * 08:30 [[gitlab:nabbegat|@nabbegat]] was approved. * 08:30 [[gitlab:plyd|@plyd]] was approved. * 07:06 "ayush8620" was rejected (pending since 2026-04-22T07:03:12.476Z). === 2026-07-21 === * 15:51 [[gitlab:panieravide|@panieravide]] was approved. * 15:33 [[gitlab:luisvilla-personal|@luisvilla-personal]] was approved. * 13:48 [[gitlab:yru|@yru]] was approved. * 13:09 [[gitlab:deevad|@deevad]] was approved. * 12:54 [[gitlab:ctdo17|@ctdo17]] was approved. * 12:36 [[gitlab:jeannenoiraud|@jeannenoiraud]] was approved. * 12:21 [[gitlab:nadiantara|@nadiantara]] was approved. * 12:18 [[gitlab:wijltcher|@wijltcher]] was approved. * 10:33 [[gitlab:nivopol|@nivopol]] was approved. * 10:30 [[gitlab:johlig|@johlig]] was approved. * 10:30 [[gitlab:majicita|@majicita]] was approved. * 10:15 [[gitlab:yongjiapeng|@yongjiapeng]] was approved. * 10:09 [[gitlab:francyskus|@francyskus]] was approved. * 09:24 [[gitlab:xanonymusx|@xanonymusx]] was approved. === 2026-07-20 === * 17:15 "leonidlednev" was rejected (pending since 2026-04-20T17:13:35.108Z). * 15:27 [[gitlab:rodrigoargenton|@rodrigoargenton]] was approved. * 05:48 "draftecho" was rejected (pending since 2026-04-20T05:48:06.953Z). === 2026-07-19 === * 12:21 [[gitlab:boivie|@boivie]] was approved. === 2026-07-18 === * 16:09 [[gitlab:pharos|@pharos]] was approved. * 15:45 [[gitlab:priyankar22|@priyankar22]] was approved. * 15:30 [[gitlab:sisyph|@sisyph]] was approved. === 2026-07-13 === * 03:45 [[gitlab:dreamyshade|@dreamyshade]] was approved. === 2026-07-12 === * 09:27 [[gitlab:smk|@smk]] was approved. === 2026-07-11 === * 14:48 "bigcereal42" was rejected (pending since 2026-04-11T14:47:40.321Z). * 12:18 "pratyushsawan" was rejected (pending since 2026-04-11T12:16:41.671Z). === 2026-07-10 === * 14:42 [[gitlab:akaza24|@akaza24]] was approved. * 12:57 [[gitlab:kormisk|@kormisk]] was approved. === 2026-07-07 === * 10:36 [[gitlab:olafjanssen|@olafjanssen]] was approved. * 06:57 "elisapoly-99" was rejected (pending since 2026-04-07T06:55:52.662Z). === 2026-07-06 === * 11:57 "ma3rouf" was rejected (pending since 2026-04-06T11:56:30.978Z). === 2026-07-02 === * 15:48 [[gitlab:tekneos|@tekneos]] was approved. === 2026-07-01 === * 15:12 [[gitlab:mugurolevy|@mugurolevy]] was approved. * 14:15 [[gitlab:vadymts1|@vadymts1]] was approved. * 09:57 "mugurolevy" was rejected (pending since 2026-04-01T09:55:19.175Z). === 2026-06-30 === * 14:27 "shivangisharma" was rejected (pending since 2026-03-31T14:26:44.932Z). === 2026-06-29 === * 19:03 [[gitlab:thisismattmiller|@thisismattmiller]] was approved. === 2026-06-28 === * 14:51 "nkwenuinadine" was rejected (pending since 2026-03-29T14:48:32.735Z). * 14:03 "vaishnavikumbhar" was rejected (pending since 2026-03-29T14:01:30.604Z). * 13:03 "stepmay" was rejected (pending since 2026-03-29T13:01:59.905Z). * 06:51 "swallroth" was rejected (pending since 2026-03-29T06:49:54.838Z). === 2026-06-26 === * 09:39 [[gitlab:lakshita28|@lakshita28]] was approved. * 07:30 [[gitlab:reeti|@reeti]] was approved. * 07:30 [[gitlab:anushka10patel|@anushka10patel]] was approved. * 07:30 "samsaesque" was rejected (pending since 2026-03-27T07:29:57.279Z). * 05:51 [[gitlab:arpithhhaaa|@arpithhhaaa]] was approved. * 05:51 [[gitlab:govindlaltl|@govindlaltl]] was approved. === 2026-06-25 === * 16:09 [[gitlab:sakuraemad|@sakuraemad]] was approved. * 07:00 "kdh8219" was rejected (pending since 2026-03-26T06:58:05.415Z). === 2026-06-24 === * 11:54 [[gitlab:sanskardubeydev|@sanskardubeydev]] was approved. * 10:09 "tanmay789q" was rejected (pending since 2026-03-25T10:07:54.602Z). === 2026-06-22 === * 19:57 [[gitlab:gouvernathor|@gouvernathor]] was approved. * 16:45 [[gitlab:lucasbelo|@lucasbelo]] was approved. * 07:15 "jason2000-cpu" was rejected (pending since 2026-03-23T07:14:09.184Z). === 2026-06-21 === * 13:18 [[gitlab:egonw|@egonw]] was approved. === 2026-06-20 === * 10:21 [[gitlab:tways2017|@tways2017]] was approved. === 2026-06-19 === * 16:06 "wilsonwang2026" was rejected (pending since 2026-03-20T16:06:05.511Z). * 04:12 [[gitlab:claudio|@claudio]] was approved. === 2026-06-18 === * 14:21 "royiswariii" was rejected (pending since 2026-03-19T14:19:16.896Z). * 13:06 [[gitlab:laurabarluzzi|@laurabarluzzi]] was approved. === 2026-06-17 === * 11:24 "adinathq8x" was rejected (pending since 2026-03-18T11:22:50.098Z). * 09:45 "nathanveritas" was rejected (pending since 2026-03-18T09:43:51.645Z). === 2026-06-15 === * 22:39 [[gitlab:mohammadhijjawi|@mohammadhijjawi]] was approved. * 14:24 "enlisar" was rejected (pending since 2026-03-16T14:23:00.109Z). * 14:06 "ayaan" was rejected (pending since 2026-03-16T14:03:31.071Z). * 10:54 "kwametech" was rejected (pending since 2026-03-16T10:54:11.083Z). === 2026-06-14 === * 17:45 [[gitlab:surajseth520|@surajseth520]] was approved. * 07:24 "malahimhaseeb" was rejected (pending since 2026-03-15T07:21:57.748Z). === 2026-06-11 === * 11:48 [[gitlab:cadddr|@cadddr]] was approved. * 11:18 "wikipiggy" was rejected (pending since 2026-03-12T11:16:09.335Z). * 07:15 [[gitlab:vesihiisi|@vesihiisi]] was approved. === 2026-06-10 === * 07:03 [[gitlab:dmiranda|@dmiranda]] was approved. === 2026-06-09 === * 14:21 [[gitlab:linkgenetic|@linkgenetic]] was approved. * 14:03 [[gitlab:sjones-ctr|@sjones-ctr]] was approved. * 12:51 [[gitlab:ekrem|@ekrem]] was approved. === 2026-06-08 === * 17:48 "jmprax" was rejected (pending since 2026-03-09T17:46:38.807Z). * 16:15 [[gitlab:ahonc|@ahonc]] was approved. * 12:57 [[gitlab:rainmonger|@rainmonger]] was approved. === 2026-06-07 === * 23:03 "shadowthewuff" was rejected (pending since 2026-03-08T23:00:53.442Z). * 11:45 "wiki-pavan" was rejected (pending since 2026-03-08T11:45:11.116Z). * 02:39 [[gitlab:launchpad|@launchpad]] was approved. === 2026-06-06 === * 14:54 "unicord" was rejected (pending since 2026-03-07T14:52:04.992Z). * 12:48 "chien" was rejected (pending since 2026-03-07T12:48:11.669Z). === 2026-06-04 === * 14:33 "only-vikas" was rejected (pending since 2026-03-05T14:32:09.186Z). === 2026-06-03 === * 15:00 [[gitlab:anafibnshahibul|@anafibnshahibul]] was approved. === 2026-06-02 === * 21:21 "mgagat" was rejected (pending since 2026-03-03T21:18:37.223Z). * 13:57 "prasunaenumarthy" was rejected (pending since 2026-03-03T13:57:14.847Z). * 05:48 [[gitlab:tmoney|@tmoney]] was approved. === 2026-06-01 === * 14:57 "vikram2101" was rejected (pending since 2026-03-02T14:54:26.550Z). * 12:03 "watshell" was rejected (pending since 2026-03-02T12:03:09.329Z). === 2026-05-29 === * 12:48 "mounikapotladurthi" was rejected (pending since 2026-02-27T12:45:38.609Z). === 2026-05-27 === * 20:00 "vinitha" was rejected (pending since 2026-02-25T19:58:43.524Z). * 16:30 "codeurluce" was rejected (pending since 2026-02-25T16:28:53.973Z). * 14:33 [[gitlab:thilio|@thilio]] was approved. === 2026-05-26 === * 12:09 "charisad" was rejected (pending since 2026-02-24T12:07:21.881Z). === 2026-05-25 === * 22:54 "ddshelto" was rejected (pending since 2026-02-23T22:52:44.427Z). * 19:51 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). * 19:48 "lakz-99" was rejected (pending since 2026-02-23T19:47:00.263Z). === 2026-05-24 === * 18:45 "jiyagupta-cs" was rejected (pending since 2026-02-22T18:43:33.176Z). === 2026-05-23 === * 13:09 [[gitlab:gauthammohanraj|@gauthammohanraj]] was approved. * 04:21 [[gitlab:staraction|@staraction]] was approved. === 2026-05-22 === * 19:03 "i-horich" was rejected (pending since 2026-02-20T19:00:43.519Z). * 01:48 "50323233" was rejected (pending since 2026-02-20T01:48:05.555Z). === 2026-05-21 === * 18:51 "kartikeyg0104" was rejected (pending since 2026-02-19T18:48:39.707Z). * 16:27 [[gitlab:renovatebot|@renovatebot]] was approved. * 16:06 [[gitlab:gkm563|@gkm563]] was approved. === 2026-05-20 === * 01:21 "beedellrokejulianlockhart" was rejected (pending since 2026-02-18T01:19:13.284Z). === 2026-05-18 === * 23:18 "wladek92" was rejected (pending since 2026-02-16T23:16:22.939Z). * 16:36 [[gitlab:effeietsanders|@effeietsanders]] was approved. === 2026-05-14 === * 21:00 [[gitlab:nehemienathan|@nehemienathan]] was approved. === 2026-05-13 === * 10:51 "ssssaaaa" was rejected (pending since 2026-02-11T10:50:36.975Z). === 2026-05-12 === * 18:06 [[gitlab:psubhashish|@psubhashish]] was approved. * 08:12 "khan" was rejected (pending since 2026-02-10T08:11:48.776Z). * 04:27 "galaxysh" was rejected (pending since 2026-02-10T04:24:59.440Z). === 2026-05-11 === * 12:18 "peterxy12" was rejected (pending since 2026-02-09T12:18:01.982Z). === 2026-05-10 === * 11:09 "yalihupokn" was rejected (pending since 2026-02-08T11:06:51.336Z). * 05:12 "wobadha" was rejected (pending since 2026-02-08T05:11:00.569Z). === 2026-05-09 === * 13:45 "bwiki" was rejected (pending since 2026-02-07T13:43:38.177Z). === 2026-05-08 === * 09:24 [[gitlab:cwilliams|@cwilliams]] was approved. === 2026-05-07 === * 14:15 "rehankhan78" was rejected (pending since 2026-02-05T14:13:37.754Z). === 2026-05-06 === * 11:24 "ari" was rejected (pending since 2026-02-04T11:24:11.760Z). * 08:09 [[gitlab:neriah|@neriah]] was approved. * 06:27 [[gitlab:status401|@status401]] was approved. === 2026-05-03 === * 09:54 [[gitlab:anilk|@anilk]] was approved. === 2026-05-02 === * 17:54 [[gitlab:sweil|@sweil]] was approved. * 17:00 [[gitlab:aoppo|@aoppo]] was approved. === 2026-05-01 === * 21:18 [[gitlab:dawalda|@dawalda]] was approved. === 2026-04-30 === * 21:42 "merohibine" was rejected (pending since 2026-01-29T21:40:00.756Z). * 20:54 [[gitlab:tfmorris|@tfmorris]] was approved. * 17:33 [[gitlab:uyen|@uyen]] was approved. * 07:39 [[gitlab:mahveotm|@mahveotm]] was approved. * 06:36 [[gitlab:leo321|@leo321]] was approved. === 2026-04-29 === * 02:27 [[gitlab:dw31415|@dw31415]] was approved. === 2026-04-28 === * 23:09 [[gitlab:dtorsani|@dtorsani]] was approved. === 2026-04-27 === * 23:42 [[gitlab:quinlan|@quinlan]] was approved. * 05:00 [[gitlab:matthewyeager|@matthewyeager]] was approved. === 2026-04-26 === * 17:36 "kuba-hajnej" was rejected (pending since 2026-01-25T17:33:32.467Z). * 13:03 "jklamo" was rejected (pending since 2026-01-25T13:02:22.936Z). === 2026-04-25 === * 20:24 [[gitlab:maldaxura|@maldaxura]] was approved. * 14:33 [[gitlab:sirtobi|@sirtobi]] was approved. * 04:18 "ice5678" was rejected (pending since 2026-01-24T04:15:30.008Z). === 2026-04-24 === * 22:06 [[gitlab:arcstur|@arcstur]] was approved. === 2026-04-22 === * 23:06 "dtorsani" was rejected (pending since 2026-01-21T23:03:25.843Z). * 22:18 [[gitlab:egezort|@egezort]] was approved. * 16:45 "nexpectarpit" was rejected (pending since 2026-01-21T16:43:21.045Z). === 2026-04-20 === * 19:15 "fitch" was rejected (pending since 2026-01-19T19:12:35.644Z). === 2026-04-19 === * 02:54 [[gitlab:neoact|@neoact]] was approved. === 2026-04-18 === * 07:06 [[gitlab:kockaadmiralac|@kockaadmiralac]] was approved. === 2026-04-17 === * 13:42 "liselot" was rejected (pending since 2026-01-16T13:39:41.909Z). === 2026-04-15 === * 17:03 "lahari" was rejected (pending since 2026-01-14T17:02:06.275Z). === 2026-04-14 === * 13:00 "surajseth520" was rejected (pending since 2026-01-13T12:59:45.906Z). * 04:51 [[gitlab:canley|@canley]] was approved. * 01:03 "bshizzle" was rejected (pending since 2026-01-13T01:00:48.120Z). === 2026-04-13 === * 15:30 [[gitlab:passimacopoulos|@passimacopoulos]] was approved. === 2026-04-11 === * 12:30 "krithash" was rejected (pending since 2026-01-10T12:27:24.731Z). === 2026-04-10 === * 15:30 "raunak1709" was rejected (pending since 2026-01-09T15:29:10.901Z). === 2026-04-07 === * 17:03 [[gitlab:supnabla|@supnabla]] was approved. === 2026-04-06 === * 20:00 [[gitlab:laerdon|@laerdon]] was approved. * 19:21 [[gitlab:ljq3|@ljq3]] was approved. === 2026-04-04 === * 11:06 "mixcc" was rejected (pending since 2026-01-03T11:03:33.922Z). === 2026-04-02 === * 05:30 [[gitlab:mbh1|@mbh1]] was approved. === 2026-04-01 === * 18:21 "yuvrajpatil17" was rejected (pending since 2025-12-31T18:20:27.991Z). * 12:12 [[gitlab:amorii0|@amorii0]] was approved. === 2026-03-31 === * 11:00 "krrishsehgal" was rejected (pending since 2025-12-30T11:00:16.384Z). === 2026-03-30 === * 15:36 [[gitlab:atsuko|@atsuko]] was approved. === 2026-03-29 === * 11:36 [[gitlab:giftcup|@giftcup]] was approved. === 2026-03-28 === * 14:51 [[gitlab:janeeva1|@janeeva1]] was approved. === 2026-03-26 === * 13:36 [[gitlab:saiphani02|@saiphani02]] was approved. * 11:48 [[gitlab:valerioboz-wmch|@valerioboz-wmch]] was approved. === 2026-03-25 === * 09:45 "quansi" was rejected (pending since 2025-12-24T09:42:13.451Z). * 02:18 [[gitlab:viztor|@viztor]] was approved. === 2026-03-24 === * 23:18 [[gitlab:maryyann|@maryyann]] was approved. * 23:01 [[gitlab:codenamenoreste|@codenamenoreste]] was approved. * 13:36 [[gitlab:marc-maillard-wmse|@marc-maillard-wmse]] was approved. * 07:39 "fred2675" was rejected (pending since 2025-12-23T07:39:11.380Z). === 2026-03-23 === * 14:51 [[gitlab:komla|@komla]] was approved. * 05:51 "lunachuck43" was rejected (pending since 2025-12-22T05:50:17.862Z). * 04:06 "reza110011" was rejected (pending since 2025-12-22T04:05:25.117Z). === 2026-03-20 === * 21:54 "mertgor" was rejected (pending since 2025-12-19T21:51:51.419Z). * 20:57 "autanmahmah" was rejected (pending since 2025-12-19T20:54:51.678Z). * 09:57 [[gitlab:nethahussain|@nethahussain]] was approved. * 09:27 [[gitlab:piewriter|@piewriter]] was approved. * 08:15 [[gitlab:dondersmooi|@dondersmooi]] was approved. === 2026-03-19 === * 21:03 "sayvhior" was rejected (pending since 2025-12-18T21:02:31.699Z). === 2026-03-18 === * 20:15 [[gitlab:martinmystere|@martinmystere]] was approved. === 2026-03-17 === * 02:51 "louperivois" was rejected (pending since 2025-12-16T02:50:48.197Z). === 2026-03-16 === * 12:54 "mokayaj857" was rejected (pending since 2025-12-15T12:53:39.015Z). * 06:18 "roamer15" was rejected (pending since 2025-12-15T06:16:38.042Z). === 2026-03-14 === * 11:12 "umaramuhammad" was rejected (pending since 2025-12-13T11:10:44.004Z). * 09:33 "akuma19" was rejected (pending since 2025-12-13T09:31:39.044Z). * 07:06 [[gitlab:syunsyunminmin|@syunsyunminmin]] was approved. === 2026-03-12 === * 20:24 [[gitlab:11wb|@11wb]] was approved. * 09:54 [[gitlab:bcxfu75k|@bcxfu75k]] was approved. === 2026-03-10 === * 09:12 [[gitlab:viktoriahillerudwmse|@viktoriahillerudwmse]] was approved. === 2026-03-06 === * 08:09 "vazhayilnewone" was rejected (pending since 2025-12-05T08:07:02.184Z). === 2026-03-04 === * 20:54 [[gitlab:elphie|@elphie]] was approved. * 11:39 "ronaldahmed" was rejected (pending since 2025-12-03T11:37:47.492Z). * 02:12 "ltslw" was rejected (pending since 2025-12-03T02:11:52.040Z). === 2026-03-02 === * 19:21 "dlopez350" was rejected (pending since 2025-12-01T19:20:38.918Z). * 18:15 [[gitlab:lsandergreen|@lsandergreen]] was approved. === 2026-03-01 === * 10:51 [[gitlab:clintacc|@clintacc]] was approved. === 2026-02-28 === * 09:24 "cardboardlamp" was rejected (pending since 2025-11-29T09:22:03.947Z). * 08:18 "wiki-pavan" was rejected (pending since 2025-11-29T08:16:24.184Z). === 2026-02-27 === * 20:45 "thisisrick25" was rejected (pending since 2025-11-28T20:42:24.454Z). === 2026-02-26 === * 13:57 "chuiimuiiofc" was rejected (pending since 2025-11-27T13:57:02.794Z). * 13:54 "steffpro" was rejected (pending since 2025-11-27T13:52:10.859Z). === 2026-02-25 === * 21:24 "abubakarhabibudayyabu" was rejected (pending since 2025-11-26T21:22:37.776Z). === 2026-02-24 === * 05:00 "playboi" was rejected (pending since 2025-11-25T05:00:30.762Z). === 2026-02-23 === * 14:00 "alph65" was rejected (pending since 2025-11-24T13:59:00.797Z). * 12:33 [[gitlab:robertsky|@robertsky]] was approved. === 2026-02-22 === * 00:30 "hp8p" was rejected (pending since 2025-11-23T00:29:24.741Z). === 2026-02-19 === * 16:45 "clayjar" was rejected (pending since 2025-11-20T16:44:48.380Z). === 2026-02-18 === * 22:18 "nexus" was rejected (pending since 2025-11-19T22:16:48.818Z). * 12:00 "bernsteinnn" was rejected (pending since 2025-11-19T11:59:04.427Z). === 2026-02-17 === * 11:36 "jason2000-cpu" was rejected (pending since 2025-11-18T11:34:00.314Z). === 2026-02-16 === * 14:54 "smaurya" was rejected (pending since 2025-11-17T14:52:06.906Z). === 2026-02-15 === * 16:51 "kra-79" was rejected (pending since 2025-11-16T16:50:41.375Z). === 2026-02-14 === * 15:15 [[gitlab:mess|@mess]] was approved. === 2026-02-13 === * 13:57 "sopalsuemae957" was rejected (pending since 2025-11-14T13:55:16.921Z). * 13:30 [[gitlab:wyslijp16-toolforge|@wyslijp16-toolforge]] was approved. === 2026-02-12 === * 16:30 "kristinagligoric" was rejected (pending since 2025-11-13T16:29:21.646Z). * 03:33 [[gitlab:anyehansen|@anyehansen]] was approved. * 02:21 [[gitlab:thejoyfultentmaker|@thejoyfultentmaker]] was approved. === 2026-02-10 === * 13:18 [[gitlab:db111|@db111]] was approved. === 2026-02-09 === * 19:06 "squirrel289" was rejected (pending since 2025-11-10T19:04:27.831Z). === 2026-02-06 === * 20:54 [[gitlab:gillux|@gillux]] was approved. * 09:09 [[gitlab:lih|@lih]] was approved. === 2026-01-31 === * 16:21 [[gitlab:taxonbot1|@taxonbot1]] was approved. === 2026-01-28 === * 14:30 [[gitlab:ademola|@ademola]] was approved. * 10:51 "watshell" was rejected (pending since 2025-10-29T10:51:01.521Z). === 2026-01-26 === * 23:06 "tavaresgmg" was rejected (pending since 2025-10-27T23:04:42.140Z). === 2026-01-25 === * 06:03 "cata" was rejected (pending since 2025-10-26T06:01:26.155Z). === 2026-01-24 === * 21:15 [[gitlab:wiegels|@wiegels]] was approved. * 06:30 [[gitlab:blaquans|@blaquans]] was approved. === 2026-01-23 === * 16:27 [[gitlab:lerickson|@lerickson]] was approved. * 10:15 "fran0035g" was rejected (pending since 2025-10-24T10:12:17.732Z). === 2026-01-22 === * 21:00 "hacksyn" was rejected (pending since 2025-10-23T20:59:15.982Z). === 2026-01-21 === * 17:30 [[gitlab:otcenas11|@otcenas11]] was approved. === 2026-01-19 === * 21:48 [[gitlab:amdrel|@amdrel]] was approved. * 04:36 "rayalexa" was rejected (pending since 2025-10-20T04:35:02.094Z). === 2026-01-18 === * 15:45 "somya" was rejected (pending since 2025-10-19T15:43:43.701Z). * 06:54 "sergg001" was rejected (pending since 2025-10-19T06:54:12.296Z). === 2026-01-16 === * 11:57 "zeejohsy" was rejected (pending since 2025-10-17T11:56:22.372Z). * 04:45 "rocky25" was rejected (pending since 2025-10-17T04:43:33.180Z). === 2026-01-15 === * 16:39 "tiisu" was rejected (pending since 2025-10-16T16:37:18.438Z). * 12:00 "noahalorwu" was rejected (pending since 2025-10-16T11:58:26.133Z). * 10:39 "prjayaiuedu" was rejected (pending since 2025-10-16T10:37:16.947Z). === 2026-01-13 === * 17:21 [[gitlab:lwilson-ctr|@lwilson-ctr]] was approved. === 2026-01-12 === * 17:03 "stagietechs" was rejected (pending since 2025-10-13T17:02:25.281Z). === 2026-01-10 === * 19:06 "keerthisr" was rejected (pending since 2025-10-11T19:05:01.758Z). === 2026-01-09 === * 20:36 "lightb" was rejected (pending since 2025-10-10T20:34:20.264Z). === 2026-01-08 === * 19:42 [[gitlab:tbodt|@tbodt]] was approved. * 13:57 [[gitlab:martynranyard|@martynranyard]] was approved. === 2026-01-07 === * 17:48 [[gitlab:santanuwiki25|@santanuwiki25]] was approved. * 14:27 "dipanshu" was rejected (pending since 2025-10-08T14:26:10.794Z). * 12:30 "adeolaadesina" was rejected (pending since 2025-10-08T12:29:49.592Z). * 09:21 "tony-kamande" was rejected (pending since 2025-10-08T09:20:28.421Z). * 06:18 "hninwuttyi" was rejected (pending since 2025-10-08T06:17:28.006Z). * 05:09 "andume" was rejected (pending since 2025-10-08T05:07:18.582Z). * 02:00 "mosope" was rejected (pending since 2025-10-08T01:59:54.800Z). * 01:15 [[gitlab:tungstalite|@tungstalite]] was approved. === 2026-01-06 === * 18:24 "leerensucher" was rejected (pending since 2025-10-07T18:21:41.253Z). * 14:54 "leonidlednev" was rejected (pending since 2025-10-07T14:53:07.273Z). * 12:57 "alexandre-tingaud" was rejected (pending since 2025-10-07T12:54:27.206Z). === 2026-01-04 === * 21:33 [[gitlab:matr1x-101|@matr1x-101]] was approved. * 15:18 "makjr" was rejected (pending since 2025-10-05T15:16:31.558Z). * 14:09 "dakshq" was rejected (pending since 2025-10-05T14:08:40.608Z). === 2026-01-03 === * 20:42 [[gitlab:apehitkey|@apehitkey]] was approved. * 18:00 [[gitlab:jeremyb|@jeremyb]] was approved. * 14:09 [[gitlab:twelephant|@twelephant]] was approved. === 2026-01-01 === * 11:30 "shellstanislav" was rejected (pending since 2025-10-02T11:29:10.150Z). === 2025-12-30 === * 19:51 "camilojdiaz" was rejected (pending since 2025-09-30T19:49:24.913Z). === 2025-12-29 === * 16:03 "zied" was rejected (pending since 2025-09-29T16:01:30.415Z). * 08:18 "rahulsidpradhan" was rejected (pending since 2025-09-29T08:17:02.849Z). === 2025-12-26 === * 09:48 "thembo42" was rejected (pending since 2025-09-26T09:45:15.033Z). === 2025-12-25 === * 14:03 "196936074751" was rejected (pending since 2025-09-25T14:02:31.367Z). === 2025-12-23 === * 16:21 "ngarnsworthy" was rejected (pending since 2025-09-23T16:20:41.211Z). === 2025-12-22 === * 12:39 "aza555" was rejected (pending since 2025-09-22T12:38:02.622Z). === 2025-12-20 === * 23:45 "saph" was rejected (pending since 2025-09-20T23:45:01.222Z). === 2025-12-19 === * 10:15 "vladdymoses" was rejected (pending since 2025-09-19T10:15:00.999Z). * 07:15 "dirtylittlepoobah" was rejected (pending since 2025-09-19T07:13:55.537Z). === 2025-12-18 === * 16:24 [[gitlab:guyfawcus|@guyfawcus]] was approved. === 2025-12-17 === * 21:39 [[gitlab:holdyourhorses|@holdyourhorses]] was approved. * 18:30 "prudencia" was rejected (pending since 2025-09-17T18:27:18.860Z). * 02:24 "lottie" was rejected (pending since 2025-09-17T02:21:21.744Z). === 2025-12-16 === * 09:39 [[gitlab:melcatherine|@melcatherine]] was approved. * 08:54 [[gitlab:leila237|@leila237]] was approved. === 2025-12-15 === * 18:27 [[gitlab:royalsailor|@royalsailor]] was approved. * 09:39 [[gitlab:olaf8940|@olaf8940]] was approved. * 09:39 "brianbybyby" was rejected (pending since 2025-09-15T09:37:45.430Z). === 2025-12-14 === * 20:21 [[gitlab:essa237|@essa237]] was approved. * 16:42 [[gitlab:bovimacoco|@bovimacoco]] was approved. === 2025-12-13 === * 21:54 "mmns21" was rejected (pending since 2025-09-13T21:52:24.017Z). * 20:33 "bugcrawler" was rejected (pending since 2025-09-13T20:31:09.211Z). === 2025-12-12 === * 14:39 "ruvchoudhary" was rejected (pending since 2025-09-12T14:36:16.167Z). * 06:54 "rezadress" was rejected (pending since 2025-09-12T06:52:21.749Z). === 2025-12-10 === * 17:30 [[gitlab:itsmoon|@itsmoon]] was approved. === 2025-12-09 === * 15:42 [[gitlab:mercy-o|@mercy-o]] was approved. === 2025-12-06 === * 16:45 "jacquesradjabu" was rejected (pending since 2025-09-06T16:45:17.969Z). * 11:27 [[gitlab:ikhitron|@ikhitron]] was approved. === 2025-12-01 === * 08:12 "halconmilenario21" was rejected (pending since 2025-09-01T08:12:10.262Z). === 2025-11-30 === * 21:06 [[gitlab:habs|@habs]] was approved. === 2025-11-29 === * 16:36 "bovimacoco" was rejected (pending since 2025-08-30T16:34:39.712Z). * 00:45 [[gitlab:jjpmaster|@jjpmaster]] was approved. === 2025-11-24 === * 10:30 "alph65" was rejected (pending since 2025-08-25T10:28:40.957Z). * 02:24 [[gitlab:yaron|@yaron]] was approved. === 2025-11-20 === * 16:06 "clayjar" was rejected (pending since 2025-08-21T16:04:54.450Z). === 2025-11-17 === * 21:09 [[gitlab:ankita97531|@ankita97531]] was approved. === 2025-11-16 === * 14:15 "commanderkefir" was rejected (pending since 2025-08-17T14:13:14.791Z). * 08:21 "rehankhan78" was rejected (pending since 2025-08-17T08:19:44.896Z). === 2025-11-15 === * 14:36 "cyberscribe" was rejected (pending since 2025-08-16T14:34:27.230Z). === 2025-11-13 === * 04:21 "waddie96" was rejected (pending since 2025-08-14T04:19:27.461Z). === 2025-11-11 === * 06:42 [[gitlab:seanhoyland|@seanhoyland]] was approved. === 2025-11-10 === * 00:06 [[gitlab:jaredblumer|@jaredblumer]] was approved. === 2025-11-09 === * 22:36 "heinxiety" was rejected (pending since 2025-08-10T22:33:12.041Z). === 2025-11-07 === * 22:00 [[gitlab:forzagreen|@forzagreen]] was approved. === 2025-11-06 === * 16:57 [[gitlab:rsilvola|@rsilvola]] was approved. === 2025-11-04 === * 21:24 [[gitlab:devdoingdev|@devdoingdev]] was approved. === 2025-11-03 === * 17:48 "joewaleed98" was rejected (pending since 2025-08-04T17:46:12.191Z). === 2025-11-01 === * 18:00 "eliasempresas" was rejected (pending since 2025-08-02T17:58:04.412Z). === 2025-10-31 === * 18:51 [[gitlab:chaoticenby|@chaoticenby]] was approved. * 04:33 "3ch310n" was rejected (pending since 2025-08-01T04:32:21.982Z). === 2025-10-30 === * 10:03 [[gitlab:tausheefhassan|@tausheefhassan]] was approved. === 2025-10-29 === * 14:54 "theap" was rejected (pending since 2025-07-30T14:52:12.066Z). === 2025-10-28 === * 06:06 [[gitlab:tanbiruzzaman|@tanbiruzzaman]] was approved. === 2025-10-27 === * 07:51 [[gitlab:jmoore111|@jmoore111]] was approved. === 2025-10-25 === * 21:09 [[gitlab:valor|@valor]] was approved. * 21:03 [[gitlab:booksmurf|@booksmurf]] was approved. * 02:48 "mystyc1" was rejected (pending since 2025-07-26T02:46:19.373Z). === 2025-10-24 === * 05:12 "aadarshmahesh" was rejected (pending since 2025-07-25T05:09:38.264Z). === 2025-10-22 === * 20:54 [[gitlab:janewanga|@janewanga]] was approved. * 17:27 "abeljeevan" was rejected (pending since 2025-07-23T17:26:46.884Z). * 16:12 "shrimpnaur" was rejected (pending since 2025-07-23T16:10:37.864Z). === 2025-10-21 === * 18:51 "jrmuizel" was rejected (pending since 2025-07-22T18:50:07.315Z). * 09:33 [[gitlab:dpogorzelski|@dpogorzelski]] was approved. === 2025-10-17 === * 13:21 [[gitlab:blegodwin|@blegodwin]] was approved. === 2025-10-16 === * 14:51 [[gitlab:bahago|@bahago]] was approved. * 14:12 "harikrishna0005" was rejected (pending since 2025-07-17T14:10:48.385Z). * 14:09 "gauthammohanraj" was rejected (pending since 2025-07-17T14:08:47.643Z). === 2025-10-15 === * 13:48 [[gitlab:adwivedii|@adwivedii]] was approved. * 13:18 [[gitlab:kimbrenekakande|@kimbrenekakande]] was approved. * 13:03 "childmnajennifer" was rejected (pending since 2025-07-16T13:01:50.236Z). * 05:06 "vssb4214" was rejected (pending since 2025-07-16T05:05:33.985Z). === 2025-10-14 === * 19:39 [[gitlab:afanyulionel|@afanyulionel]] was approved. * 15:33 [[gitlab:sadrettin|@sadrettin]] was approved. * 14:18 [[gitlab:tmwyk|@tmwyk]] was approved. * 08:42 "yasu0796" was rejected (pending since 2025-07-15T08:41:26.453Z). === 2025-10-13 === * 16:09 [[gitlab:atlas0007|@atlas0007]] was approved. === 2025-10-11 === * 17:42 [[gitlab:techwizzie|@techwizzie]] was approved. === 2025-10-10 === * 19:03 [[gitlab:miiswom|@miiswom]] was approved. * 16:06 [[gitlab:ninatakang|@ninatakang]] was approved. === 2025-10-09 === * 15:42 [[gitlab:jaykaneki|@jaykaneki]] was approved. * 14:21 [[gitlab:lebogang|@lebogang]] was approved. * 14:15 [[gitlab:kimondorose|@kimondorose]] was approved. * 13:48 [[gitlab:joyakinyi|@joyakinyi]] was approved. * 13:48 [[gitlab:dikshyashahi|@dikshyashahi]] was approved. * 13:45 [[gitlab:obediobadiah|@obediobadiah]] was approved. * 13:45 [[gitlab:system625|@system625]] was approved. * 13:45 [[gitlab:rolalove|@rolalove]] was approved. * 13:39 [[gitlab:olatundeawo|@olatundeawo]] was approved. * 13:36 [[gitlab:danielchristlight|@danielchristlight]] was approved. * 13:36 [[gitlab:dipanshu1223|@dipanshu1223]] was approved. * 13:36 [[gitlab:aradhya|@aradhya]] was approved. * 09:57 "bognd" was rejected (pending since 2025-07-10T09:55:48.661Z). === 2025-10-08 === * 23:36 [[gitlab:sopzy|@sopzy]] was approved. * 23:03 [[gitlab:oluwatumininu|@oluwatumininu]] was approved. * 19:39 [[gitlab:levon003|@levon003]] was approved. * 15:24 [[gitlab:ritika-bhambri11|@ritika-bhambri11]] was approved. * 13:45 [[gitlab:anbanguyen|@anbanguyen]] was approved. * 13:36 [[gitlab:chumzine|@chumzine]] was approved. * 13:27 [[gitlab:shr0x-ya|@shr0x-ya]] was approved. * 12:45 [[gitlab:nurahwakili|@nurahwakili]] was approved. * 03:42 "nazhiba" was rejected (pending since 2025-07-09T03:40:12.625Z). * 02:12 "mafennel" was rejected (pending since 2025-07-09T02:11:40.598Z). === 2025-10-07 === * 22:54 [[gitlab:olusegunfaj|@olusegunfaj]] was approved. * 21:30 [[gitlab:rona|@rona]] was approved. * 21:09 [[gitlab:sandijigs|@sandijigs]] was approved. * 13:36 "xisbajao" was rejected (pending since 2025-07-08T13:33:35.018Z). * 01:36 "areczek94" was rejected (pending since 2025-07-08T01:35:40.633Z). === 2025-10-06 === * 19:21 "wmcarter2017" was rejected (pending since 2025-07-07T19:21:12.899Z). === 2025-10-05 === * 14:15 "meetmendapara" was rejected (pending since 2025-07-06T14:14:16.726Z). === 2025-10-04 === * 20:51 "nftbaee" was rejected (pending since 2025-07-05T20:50:57.688Z). === 2025-10-03 === * 06:12 [[gitlab:javiermonton|@javiermonton]] was approved. === 2025-10-02 === * 20:15 "talaqalotaibipmp" was rejected (pending since 2025-07-03T20:13:05.164Z). === 2025-10-01 === * 10:54 "bjensen" was rejected (pending since 2025-07-02T10:53:46.574Z). * 02:45 "kowal1984" was rejected (pending since 2025-07-02T02:44:56.946Z). === 2025-09-30 === * 21:21 [[gitlab:kavaljeetsingh|@kavaljeetsingh]] was approved. * 00:24 "adium" was rejected (pending since 2025-07-01T00:23:43.807Z). === 2025-09-28 === * 08:54 [[gitlab:pexerik|@pexerik]] was approved. === 2025-09-27 === * 13:57 [[gitlab:rubahhitamvukova|@rubahhitamvukova]] was approved. === 2025-09-26 === * 16:57 "algorithmic" was rejected (pending since 2025-06-27T16:56:17.480Z). * 13:54 [[gitlab:shadabgdg|@shadabgdg]] was approved. * 13:12 [[gitlab:spushpit|@spushpit]] was approved. === 2025-09-20 === * 14:06 "bwiki" was rejected (pending since 2025-06-21T13:59:14.749Z). === 2025-09-16 === * 05:39 [[gitlab:deepchirp|@deepchirp]] was approved. === 2025-09-15 === * 22:00 [[gitlab:noisk8|@noisk8]] was approved. * 11:03 "ahonc" was rejected (pending since 2025-06-16T11:00:54.843Z). === 2025-09-13 === * 18:24 "a-ssh22" was rejected (pending since 2025-06-14T18:23:33.937Z). * 12:36 [[gitlab:rajashreetalukdar|@rajashreetalukdar]] was approved. * 00:45 [[gitlab:sumitsurai|@sumitsurai]] was approved. === 2025-09-12 === * 17:12 [[gitlab:suyash23|@suyash23]] was approved. * 00:46 "remotetravel" was rejected (pending since 2025-06-13T00:44:08.171Z). === 2025-09-10 === * 21:09 "jancborchardt" was rejected (pending since 2025-06-11T21:06:30.759Z). === 2025-09-09 === * 17:03 [[gitlab:vwf|@vwf]] was approved. * 06:36 [[gitlab:cactusisme|@cactusisme]] was approved. === 2025-09-08 === * 18:09 "birushandegeya" was rejected (pending since 2025-06-09T18:08:00.087Z). * 16:27 "ngarnsworthy" was rejected (pending since 2025-06-09T16:24:37.213Z). * 12:33 "zolgoyo" was rejected (pending since 2025-06-09T12:31:34.199Z). === 2025-09-06 === * 23:09 [[gitlab:jaishsingh913|@jaishsingh913]] was approved. === 2025-09-05 === * 21:45 [[gitlab:sakshi2|@sakshi2]] was approved. * 20:42 "abdukhaliq1" was rejected (pending since 2025-06-06T20:40:42.023Z). * 14:27 "beubsamy" was rejected (pending since 2025-06-06T14:27:06.781Z). === 2025-09-04 === * 23:27 "sdhehua" was rejected (pending since 2025-06-05T23:24:45.777Z). * 19:00 [[gitlab:perry|@perry]] was approved. * 11:24 "saintwolf" was rejected (pending since 2025-06-05T11:21:20.176Z). === 2025-09-02 === * 05:48 [[gitlab:aliu|@aliu]] was approved. === 2025-08-29 === * 13:30 "kksurendran066" was rejected (pending since 2025-05-30T13:27:48.755Z). === 2025-08-28 === * 22:18 "tauraamuix" was rejected (pending since 2025-05-29T22:16:08.228Z). === 2025-08-26 === * 19:03 [[gitlab:dikkulah|@dikkulah]] was approved. === 2025-08-22 === * 23:51 [[gitlab:khoroshun_mike|@khoroshun_mike]] was approved. === 2025-08-21 === * 07:39 [[gitlab:yuka|@yuka]] was approved. === 2025-08-19 === * 07:48 [[gitlab:zhaofjx|@zhaofjx]] was approved. === 2025-08-17 === * 14:27 "madhan13k" was rejected (pending since 2025-05-18T14:26:08.973Z). === 2025-08-15 === * 10:15 "mohammed_abukhadra" was rejected (pending since 2025-05-16T10:14:48.403Z). === 2025-08-11 === * 11:48 "hmmyesbro" was rejected (pending since 2025-05-12T11:45:24.350Z). === 2025-08-10 === * 13:15 [[gitlab:dactyl|@dactyl]] was approved. === 2025-08-09 === * 04:39 "xxxx100000" was rejected (pending since 2025-05-10T04:37:44.949Z). === 2025-08-08 === * 14:33 [[gitlab:josefanthony|@josefanthony]] was approved. === 2025-08-07 === * 23:42 [[gitlab:robins7|@robins7]] was approved. * 21:42 [[gitlab:pols12|@pols12]] was approved. * 17:15 "sbronson" was rejected (pending since 2025-05-08T17:15:08.834Z). * 14:57 [[gitlab:alvindulle|@alvindulle]] was approved. * 14:45 [[gitlab:xentos|@xentos]] was approved. * 06:27 "jamesboste" was rejected (pending since 2025-05-08T06:25:14.793Z). * 03:57 "ysun" was rejected (pending since 2025-05-08T03:55:07.348Z). === 2025-08-06 === * 21:51 "pols12" was rejected (pending since 2025-05-07T21:49:13.598Z). * 01:51 "okeamah" was rejected (pending since 2025-05-07T01:48:50.114Z). === 2025-08-05 === * 09:15 "mobashir-2013" was rejected (pending since 2025-05-06T09:14:24.069Z). === 2025-08-01 === * 08:00 "douginamug" was rejected (pending since 2025-05-02T07:57:38.317Z). === 2025-07-31 === * 02:30 [[gitlab:ads|@ads]] was approved. === 2025-07-27 === * 13:15 "mrico2703" was rejected (pending since 2025-04-27T13:13:12.346Z). * 10:17 [[gitlab:josephfrancis12|@josephfrancis12]] was approved. * 10:17 [[gitlab:fuzzew|@fuzzew]] was approved. * 05:57 [[gitlab:biscuitbobby|@biscuitbobby]] was approved. * 05:48 [[gitlab:ecoholic|@ecoholic]] was approved. === 2025-07-26 === * 11:48 [[gitlab:chimnayyyy|@chimnayyyy]] was approved. * 11:48 [[gitlab:alwinalbert|@alwinalbert]] was approved. * 11:48 [[gitlab:hridyakk|@hridyakk]] was approved. * 11:45 [[gitlab:gaurigupta21|@gaurigupta21]] was approved. * 11:45 [[gitlab:binetaa|@binetaa]] was approved. * 10:21 [[gitlab:jyothikat22|@jyothikat22]] was approved. * 10:21 [[gitlab:zobotrombie|@zobotrombie]] was approved. * 10:21 [[gitlab:flykrth|@flykrth]] was approved. * 10:21 [[gitlab:mehrinshamim|@mehrinshamim]] was approved. * 10:21 [[gitlab:aadhi13|@aadhi13]] was approved. * 10:21 [[gitlab:malavikam05|@malavikam05]] was approved. * 10:18 [[gitlab:nf609|@nf609]] was approved. * 05:48 [[gitlab:nazalnihad|@nazalnihad]] was approved. * 05:48 [[gitlab:naveen28204280|@naveen28204280]] was approved. === 2025-07-25 === * 09:49 [[gitlab:kasyap9|@kasyap9]] was approved. * 09:30 [[gitlab:swayamagrahari|@swayamagrahari]] was approved. === 2025-07-24 === * 19:36 [[gitlab:madutgn|@madutgn]] was approved. === 2025-07-23 === * 20:09 [[gitlab:somerandomdeveloper|@somerandomdeveloper]] was approved. === 2025-07-22 === * 00:15 [[gitlab:iagoqnsi|@iagoqnsi]] was approved. === 2025-07-21 === * 17:30 [[gitlab:asadiqui|@asadiqui]] was approved. * 16:39 [[gitlab:tryvix1509|@tryvix1509]] was approved. * 04:27 [[gitlab:damian|@damian]] was approved. === 2025-07-20 === * 09:42 "mike-khoroshun" was rejected (pending since 2025-04-20T09:42:22.732Z). === 2025-07-17 === * 17:57 [[gitlab:haroldkrabs|@haroldkrabs]] was approved. * 13:45 [[gitlab:envlh|@envlh]] was approved. === 2025-07-14 === * 10:24 [[gitlab:missguru|@missguru]] was approved. * 00:57 "clarfonthey" was rejected (pending since 2025-04-14T00:56:32.626Z). === 2025-07-13 === * 01:01 [[gitlab:l235|@l235]] was approved. === 2025-07-11 === * 03:06 "rodavlas" was rejected (pending since 2025-04-11T03:05:45.590Z). === 2025-07-06 === * 00:09 "lakasa" was rejected (pending since 2025-04-06T00:06:28.469Z). === 2025-07-05 === * 21:54 "ctrlzvi" was rejected (pending since 2025-04-05T21:54:12.542Z). * 14:30 "aminualiyu" was rejected (pending since 2025-04-05T14:27:22.617Z). === 2025-07-04 === * 03:15 [[gitlab:galstar|@galstar]] was approved. === 2025-07-02 === * 11:27 "vicolas11" was rejected (pending since 2025-04-02T11:25:12.682Z). === 2025-06-29 === * 23:12 "naomi723" was rejected (pending since 2025-03-30T23:09:24.630Z). === 2025-06-28 === * 16:21 "mudeh2372" was rejected (pending since 2025-03-29T16:18:27.057Z). === 2025-06-27 === * 23:18 "rony143" was rejected (pending since 2025-03-28T23:16:13.671Z). * 22:21 [[gitlab:rluts|@rluts]] was approved. === 2025-06-26 === * 13:54 "creativegurus" was rejected (pending since 2025-03-27T13:52:41.706Z). === 2025-06-24 === * 17:42 [[gitlab:devjadiya|@devjadiya]] was approved. * 14:00 "dominic-r" was rejected (pending since 2025-03-25T14:00:07.307Z). === 2025-06-21 === * 00:48 [[gitlab:vriaa|@vriaa]] was approved. === 2025-06-18 === * 15:21 "ayushkhati1" was rejected (pending since 2025-03-19T15:18:50.062Z). === 2025-06-17 === * 20:45 "chiomavero" was rejected (pending since 2025-03-18T20:44:13.967Z). * 00:27 [[gitlab:eggroll97|@eggroll97]] was approved. === 2025-06-14 === * 20:57 "volvox" was rejected (pending since 2025-03-15T20:56:34.018Z). === 2025-06-13 === * 16:09 [[gitlab:supergrey|@supergrey]] was approved. * 11:03 "chqaz" was rejected (pending since 2025-03-14T11:01:09.600Z). * 10:24 [[gitlab:slong-wmf|@slong-wmf]] was approved. * 10:15 "hearvox" was rejected (pending since 2025-03-14T10:13:13.112Z). === 2025-06-12 === * 15:18 "jlam" was rejected (pending since 2025-03-13T15:17:54.099Z). === 2025-06-09 === * 20:48 "dipanjansengupta" was rejected (pending since 2025-03-10T20:48:03.545Z). * 19:27 [[gitlab:reggycelly|@reggycelly]] was approved. * 14:51 "arendpieter" was rejected (pending since 2025-03-10T14:51:01.445Z). * 13:21 [[gitlab:greenreaper|@greenreaper]] was approved. * 09:33 [[gitlab:mmta|@mmta]] was approved. * 08:03 "a-ssh22" was rejected (pending since 2025-03-10T08:03:08.111Z). === 2025-06-08 === * 21:06 "mm-episodenlistedlvaupdater" was rejected (pending since 2025-03-09T21:04:06.323Z). === 2025-06-06 === * 11:06 [[gitlab:olea|@olea]] was approved. === 2025-06-05 === * 20:33 [[gitlab:encodedwp|@encodedwp]] was approved. * 15:00 [[gitlab:toluayo|@toluayo]] was approved. * 13:51 [[gitlab:arnold_lup|@arnold_lup]] was approved. * 11:54 "sdhehua" was rejected (pending since 2025-03-06T11:51:48.241Z). === 2025-06-03 === * 21:27 [[gitlab:wewakey|@wewakey]] was approved. * 12:36 "hunsimon2" was rejected (pending since 2025-03-04T12:34:56.520Z). * 11:54 "hunsimon" was rejected (pending since 2025-03-04T11:53:54.652Z). === 2025-06-02 === * 12:01 [[gitlab:jaimedes|@jaimedes]] was approved. === 2025-05-30 === * 18:00 "sathvik9105" was rejected (pending since 2025-02-28T17:59:42.867Z). * 11:21 [[gitlab:tonythomas01|@tonythomas01]] was approved. * 10:06 [[gitlab:gpsleo|@gpsleo]] was approved. === 2025-05-29 === * 22:12 [[gitlab:codynguyen1116|@codynguyen1116]] was approved. === 2025-05-28 === * 02:57 [[gitlab:saper|@saper]] was approved. === 2025-05-27 === * 21:06 [[gitlab:mohammed_qays|@mohammed_qays]] was approved. * 15:33 "satanluimm" was rejected (pending since 2025-02-25T15:32:48.101Z). === 2025-05-26 === * 23:57 "seyedali220" was rejected (pending since 2025-02-24T23:56:17.621Z). === 2025-05-21 === * 11:12 [[gitlab:guilherme|@guilherme]] was approved. === 2025-05-19 === * 13:24 [[gitlab:emojiwiki|@emojiwiki]] was approved. === 2025-05-18 === * 00:00 "xidme" was rejected (pending since 2025-02-15T23:58:56.796Z). === 2025-05-17 === * 02:39 "kdh8219" was rejected (pending since 2025-02-15T02:36:32.237Z). === 2025-05-16 === * 15:09 [[gitlab:maxbinderwmf|@maxbinderwmf]] was approved. === 2025-05-15 === * 04:30 "inspectorzer0" was rejected (pending since 2025-02-13T04:27:33.179Z). === 2025-05-14 === * 17:42 [[gitlab:llugo|@llugo]] was approved. === 2025-05-13 === * 20:18 "mmta" was rejected (pending since 2025-02-11T20:17:23.407Z). === 2025-05-11 === * 20:51 "jad" was rejected (pending since 2025-02-09T20:49:07.333Z). * 17:54 "nishchalsundan" was rejected (pending since 2025-02-09T17:52:25.761Z). * 16:39 "mohammed_abukhadra" was rejected (pending since 2025-02-09T16:39:03.730Z). === 2025-05-09 === * 09:12 [[gitlab:sirchanmp|@sirchanmp]] was approved. === 2025-05-08 === * 08:18 [[gitlab:mengeditch|@mengeditch]] was approved. === 2025-05-07 === * 03:45 "xluffy" was rejected (pending since 2025-02-05T03:45:14.181Z). === 2025-05-06 === * 16:54 "punhaniabhishek" was rejected (pending since 2025-02-04T16:53:50.758Z). * 09:36 [[gitlab:bmartinezcalvo|@bmartinezcalvo]] was approved. === 2025-05-02 === * 12:24 [[gitlab:tohaomg|@tohaomg]] was approved. * 11:48 [[gitlab:mavrikant|@mavrikant]] was approved. * 11:45 [[gitlab:daanvr|@daanvr]] was approved. === 2025-05-01 === * 09:09 "mjoerg" was rejected (pending since 2025-01-30T09:09:04.204Z). === 2025-04-30 === * 23:06 "sanskardubey" was rejected (pending since 2025-01-29T23:03:25.489Z). === 2025-04-29 === * 16:00 "geyslein" was rejected (pending since 2025-01-28T16:00:01.510Z). === 2025-04-26 === * 09:30 "anjali9027" was rejected (pending since 2025-01-25T09:28:07.064Z). === 2025-04-25 === * 18:00 "salahhazaa" was rejected (pending since 2025-01-24T17:58:30.030Z). * 15:15 [[gitlab:yiming|@yiming]] was approved. * 02:06 "mrchanmp" was rejected (pending since 2025-01-24T02:03:58.308Z). === 2025-04-23 === * 17:03 "rj2904" was rejected (pending since 2025-01-22T17:03:11.207Z). * 14:21 "nischay33" was rejected (pending since 2025-01-22T14:19:21.081Z). === 2025-04-22 === * 19:27 "dj80" was rejected (pending since 2025-01-21T19:25:28.498Z). * 14:30 [[gitlab:kaimamin|@kaimamin]] was approved. * 09:57 "debo" was rejected (pending since 2025-01-21T09:54:47.955Z). === 2025-04-21 === * 12:24 "unshell" was rejected (pending since 2025-01-20T12:21:59.686Z). === 2025-04-18 === * 15:06 [[gitlab:spartanarbinger|@spartanarbinger]] was approved. === 2025-04-16 === * 03:09 "dewey" was rejected (pending since 2025-01-15T03:06:17.488Z). === 2025-04-15 === * 19:45 "emdadul" was rejected (pending since 2025-01-14T19:42:29.285Z). === 2025-04-14 === * 06:45 [[gitlab:bcampbell804|@bcampbell804]] was approved. === 2025-04-11 === * 06:27 [[gitlab:jvanderhoop|@jvanderhoop]] was approved. === 2025-04-10 === * 04:12 "bhai420" was rejected (pending since 2025-01-09T04:10:29.430Z). === 2025-04-09 === * 05:03 "austinvarshney" was rejected (pending since 2025-01-08T05:02:34.175Z). === 2025-04-06 === * 15:36 [[gitlab:elph|@elph]] was approved. === 2025-04-02 === * 10:33 [[gitlab:ozge|@ozge]] was approved. === 2025-03-31 === * 20:15 "demandkey" was rejected (pending since 2024-12-30T20:14:23.096Z). * 15:18 [[gitlab:danyya|@danyya]] was approved. === 2025-03-28 === * 15:54 [[gitlab:rutsavi09|@rutsavi09]] was approved. * 15:54 [[gitlab:ilanen1|@ilanen1]] was approved. === 2025-03-25 === * 19:27 [[gitlab:irfo|@irfo]] was approved. * 11:54 [[gitlab:kmontalva-wmf|@kmontalva-wmf]] was approved. * 04:33 [[gitlab:paul26|@paul26]] was approved. * 04:18 "as1100k" was rejected (pending since 2024-12-24T04:18:06.813Z). === 2025-03-24 === * 11:33 "amzadkhankk" was rejected (pending since 2024-12-23T11:33:14.176Z). === 2025-03-23 === * 12:24 "wolfdo" was rejected (pending since 2024-12-22T12:23:35.056Z). === 2025-03-22 === * 09:45 [[gitlab:fjmustak|@fjmustak]] was approved. === 2025-03-20 === * 18:42 "sathishkokila" was rejected (pending since 2024-12-19T18:39:35.161Z). * 17:03 [[gitlab:alien4444|@alien4444]] was approved. * 15:27 [[gitlab:davidcoronel|@davidcoronel]] was approved. === 2025-03-19 === * 22:57 [[gitlab:r1f4t|@r1f4t]] was approved. * 19:03 "daniel24ps" was rejected (pending since 2024-12-18T19:00:21.249Z). * 14:18 [[gitlab:beepbooppenguin|@beepbooppenguin]] was approved. === 2025-03-18 === * 17:48 "rahulkundu1209" was rejected (pending since 2024-12-17T17:46:41.936Z). * 08:15 "kirtisikka972" was rejected (pending since 2024-12-17T08:13:25.487Z). === 2025-03-15 === * 13:30 "tulspal_sidhu" was rejected (pending since 2024-12-14T13:29:10.606Z). * 01:39 "peacedeadc" was rejected (pending since 2024-12-14T01:37:36.579Z). === 2025-03-14 === * 03:51 [[gitlab:chuckthebuck|@chuckthebuck]] was approved. * 02:33 "yxngtrtxll" was rejected (pending since 2024-12-13T02:31:51.658Z). === 2025-03-13 === * 14:36 [[gitlab:iccander|@iccander]] was approved. === 2025-03-12 === * 23:21 "jokerchic36" was rejected (pending since 2024-12-11T23:21:00.670Z). * 15:30 [[gitlab:naomi|@naomi]] was approved. * 15:27 [[gitlab:cobi|@cobi]] was approved. === 2025-03-11 === * 12:42 "mohitvermaxx" was rejected (pending since 2024-12-10T12:40:56.967Z). === 2025-03-10 === * 16:51 [[gitlab:nanona15dobato|@nanona15dobato]] was approved. === 2025-03-09 === * 22:39 [[gitlab:jonkolbert|@jonkolbert]] was approved. * 20:45 [[gitlab:urbanecmtest2|@urbanecmtest2]] was approved. === 2025-03-07 === * 16:54 [[gitlab:hswan|@hswan]] was approved. * 14:42 [[gitlab:atitkov|@atitkov]] was approved. * 00:42 [[gitlab:infrastruktur|@infrastruktur]] was approved. === 2025-03-06 === * 17:21 "johnmann" was rejected (pending since 2024-12-05T17:19:24.995Z). === 2025-03-05 === * 07:33 [[gitlab:monx9494|@monx9494]] was approved. === 2025-03-02 === * 21:21 "paul26" was rejected (pending since 2024-12-01T21:20:19.681Z). === 2025-03-01 === * 19:15 [[gitlab:izno|@izno]] was approved. * 12:45 [[gitlab:nyerho|@nyerho]] was approved. === 2025-02-28 === * 18:27 [[gitlab:chuckonwumelu|@chuckonwumelu]] was approved. * 13:09 "ashwinpraveengo" was rejected (pending since 2024-11-29T13:07:47.240Z). * 00:18 "eduardoaugusto" was rejected (pending since 2024-11-29T00:17:43.372Z). === 2025-02-27 === * 20:39 "volkanurl" was rejected (pending since 2024-11-28T20:37:18.101Z). === 2025-02-24 === * 21:15 [[gitlab:feeglgeef|@feeglgeef]] was approved. * 20:18 [[gitlab:piaanalysis2|@piaanalysis2]] was approved. * 19:06 [[gitlab:dhardy|@dhardy]] was approved. === 2025-02-22 === * 19:27 [[gitlab:owuh|@owuh]] was approved. === 2025-02-19 === * 16:06 [[gitlab:artemkloko|@artemkloko]] was approved. * 13:03 [[gitlab:jgafnea|@jgafnea]] was approved. === 2025-02-17 === * 16:33 [[gitlab:asmartkitten|@asmartkitten]] was approved. === 2025-02-16 === * 19:12 "gaurigupta21" was rejected (pending since 2024-11-17T19:11:07.416Z). === 2025-02-15 === * 01:18 [[gitlab:mediawiki-quickstart-ci|@mediawiki-quickstart-ci]] was approved. === 2025-02-14 === * 15:21 "nathanbnm" was rejected (pending since 2024-11-15T15:18:19.632Z). === 2025-02-13 === * 16:45 [[gitlab:priyanshuchahal|@priyanshuchahal]] was approved. * 16:42 [[gitlab:ajhalili2006|@ajhalili2006]] was approved. === 2025-02-12 === * 23:21 "monkeypatch999" was rejected (pending since 2024-11-13T23:20:38.398Z). * 06:36 [[gitlab:jainlakshita28|@jainlakshita28]] was approved. === 2025-02-11 === * 19:27 [[gitlab:matthewsm2|@matthewsm2]] was approved. === 2025-02-09 === * 16:15 "mohammed_abukhadra" was rejected (pending since 2024-11-10T16:15:18.361Z). === 2025-02-07 === * 21:33 "brennan" was rejected (pending since 2024-11-08T21:31:07.351Z). === 2025-02-06 === * 08:24 "mmta" was rejected (pending since 2024-11-07T08:22:36.724Z). * 06:21 [[gitlab:bunnypranav|@bunnypranav]] was approved. === 2025-02-05 === * 22:39 "chrissteinchen" was rejected (pending since 2024-11-06T22:38:16.673Z). === 2025-02-03 === * 07:45 "edriiic" was rejected (pending since 2024-11-04T07:44:46.849Z). * 01:12 "geppy" was rejected (pending since 2024-11-04T01:10:48.710Z). === 2025-02-02 === * 13:18 "funa-enpitu" was rejected (pending since 2024-11-03T13:15:46.065Z). === 2025-01-31 === * 23:42 "nfontes" was rejected (pending since 2024-11-01T23:39:41.755Z). * 22:51 "sbronson" was rejected (pending since 2024-11-01T22:50:31.871Z). * 00:42 [[gitlab:farid|@farid]] was approved. === 2025-01-27 === * 08:15 [[gitlab:eliza189|@eliza189]] was approved. === 2025-01-25 === * 09:51 [[gitlab:pamputt|@pamputt]] was approved. === 2025-01-23 === * 14:30 [[gitlab:lubianat|@lubianat]] was approved. * 11:45 [[gitlab:bootsa|@bootsa]] was approved. === 2025-01-21 === * 05:09 "niko" was rejected (pending since 2024-07-21T16:10:01.377Z). * 05:09 "thawizkid369777" was rejected (pending since 2024-07-18T17:42:44.493Z). * 05:09 "sarthaksingh2" was rejected (pending since 2024-07-10T11:31:30.470Z). * 05:09 "shriyakt" was rejected (pending since 2024-07-06T04:54:10.248Z). * 05:09 "akshaya" was rejected (pending since 2024-07-06T04:04:51.488Z). * 05:09 "alaka03aj" was rejected (pending since 2024-07-05T18:01:54.876Z). * 05:09 "sulochanaviji-5049" was rejected (pending since 2024-07-01T05:58:00.427Z). * 05:09 "nayanjnath" was rejected (pending since 2024-07-01T02:51:57.405Z). * 05:09 "sd44" was rejected (pending since 2024-06-30T04:28:51.436Z). * 05:09 "metavalent" was rejected (pending since 2024-06-29T01:37:14.210Z). * 05:09 "wicloudx" was rejected (pending since 2024-06-28T11:51:23.335Z). * 05:09 "debo" was rejected (pending since 2024-06-28T01:44:59.845Z). * 05:09 "bwiki" was rejected (pending since 2024-06-23T14:15:38.032Z). * 05:09 "toprak" was rejected (pending since 2024-06-23T11:35:50.819Z). * 05:09 "iristeller" was rejected (pending since 2024-06-14T20:53:48.959Z). * 05:09 "jcolvin" was rejected (pending since 2024-06-12T17:29:01.238Z). * 05:09 "kalyan" was rejected (pending since 2024-06-07T07:52:46.993Z). * 05:09 "bluecrystal" was rejected (pending since 2024-06-06T19:16:20.107Z). * 05:09 "iftttrohit" was rejected (pending since 2024-06-04T12:08:50.818Z). * 05:09 "pogpotato" was rejected (pending since 2024-06-03T17:58:21.684Z). * 05:09 "cptlausebaer" was rejected (pending since 2024-05-31T18:53:27.692Z). * 05:09 "hdevine825" was rejected (pending since 2024-05-31T17:04:18.279Z). * 05:09 "anaghaa18" was rejected (pending since 2024-05-25T19:14:31.803Z). * 05:09 "atharvanair04" was rejected (pending since 2024-05-25T14:24:52.825Z). * 05:09 "anasvemmully" was rejected (pending since 2024-05-25T06:10:27.261Z). * 05:09 "abhinavmohandas" was rejected (pending since 2024-05-25T06:05:24.825Z). * 05:09 "kksurendran06" was rejected (pending since 2024-05-25T06:04:38.082Z). * 05:09 "albertmarshall8896" was rejected (pending since 2024-05-23T09:32:05.462Z). * 05:09 "akellison" was rejected (pending since 2024-05-17T02:07:24.229Z). * 05:09 "mainowill" was rejected (pending since 2024-04-16T23:30:33.881Z). * 05:09 "bzhqc" was rejected (pending since 2024-04-16T19:50:38.676Z). * 05:09 "safan41" was rejected (pending since 2024-04-16T03:34:48.942Z). * 05:09 "mgagat" was rejected (pending since 2024-04-16T03:21:51.764Z). * 05:09 "okeamah" was rejected (pending since 2024-04-16T02:49:00.143Z). * 05:09 "xuhao61" was rejected (pending since 2024-04-15T23:45:09.083Z). * 04:47 "cybel" was rejected (pending since 2024-04-15T06:46:35.791Z). === 2025-01-20 === * 14:33 [[gitlab:your1|@your1]] was approved. === 2025-01-18 === * 10:09 [[gitlab:galrach600|@galrach600]] was approved. * 02:51 [[gitlab:blankeclair|@blankeclair]] was approved. === 2025-01-17 === * 13:57 [[gitlab:dsantamaria|@dsantamaria]] was approved. === 2025-01-15 === * 17:12 [[gitlab:smartse|@smartse]] was approved. === 2025-01-14 === * 17:03 [[gitlab:naorleizer|@naorleizer]] was approved. === 2025-01-13 === * 02:45 [[gitlab:wolf20482|@wolf20482]] was approved. === 2025-01-12 === * 17:45 [[gitlab:tamzin|@tamzin]] was approved. === 2025-01-11 === * 15:24 [[gitlab:bargioni|@bargioni]] was approved. * 14:30 [[gitlab:salelya|@salelya]] was approved. * 10:15 [[gitlab:malakatshy|@malakatshy]] was approved. * 05:21 [[gitlab:newmcpee|@newmcpee]] was approved. === 2025-01-09 === * 15:30 [[gitlab:gkyziridis|@gkyziridis]] was approved. === 2025-01-08 === * 16:21 [[gitlab:ukrface|@ukrface]] was approved. === 2024-12-28 === * 03:27 [[gitlab:twonum|@twonum]] was approved. === 2024-12-25 === * 06:09 [[gitlab:harsv567|@harsv567]] was approved. === 2024-12-21 === * 11:24 [[gitlab:amutha2002|@amutha2002]] was approved. === 2024-12-20 === * 19:51 [[gitlab:hridyeshgupta|@hridyeshgupta]] was approved. * 10:00 [[gitlab:ro-shines|@ro-shines]] was approved. * 08:09 [[gitlab:kesharwaniarpita|@kesharwaniarpita]] was approved. === 2024-12-18 === * 14:45 [[gitlab:soylacarli|@soylacarli]] was approved. === 2024-12-16 === * 20:33 [[gitlab:aleyasiddika1|@aleyasiddika1]] was approved. === 2024-12-15 === * 07:33 [[gitlab:abhishek02bhardwaj|@abhishek02bhardwaj]] was approved. === 2024-12-13 === * 13:18 [[gitlab:ashmitabathre204|@ashmitabathre204]] was approved. === 2024-12-10 === * 06:39 [[gitlab:ginaan|@ginaan]] was approved. === 2024-12-09 === * 05:45 [[gitlab:kallinavya|@kallinavya]] was approved. * 00:54 [[gitlab:viserion-7|@viserion-7]] was approved. === 2024-12-08 === * 17:27 [[gitlab:wargo|@wargo]] was approved. === 2024-12-05 === * 11:15 [[gitlab:ranjithraj|@ranjithraj]] was approved. === 2024-12-02 === * 21:21 [[gitlab:a930913|@a930913]] was approved. === 2024-12-01 === * 02:39 [[gitlab:kingchristlike1|@kingchristlike1]] was approved. === 2024-11-21 === * 13:45 [[gitlab:sascha|@sascha]] was approved. === 2024-11-19 === * 16:36 [[gitlab:jly|@jly]] was approved. === 2024-11-15 === * 02:54 [[gitlab:danielyepezgarces|@danielyepezgarces]] was approved. === 2024-11-14 === * 14:15 [[gitlab:stimoroll|@stimoroll]] was approved. === 2024-11-09 === * 17:15 [[gitlab:f4udeveloper|@f4udeveloper]] was approved. === 2024-11-07 === * 19:15 [[gitlab:zulf|@zulf]] was approved. * 05:33 [[gitlab:hassanamin|@hassanamin]] was approved. === 2024-11-06 === * 19:39 [[gitlab:daniuu|@daniuu]] was approved. * 00:18 [[gitlab:rlopez-wmf|@rlopez-wmf]] was approved. === 2024-10-09 === * 14:45 [[gitlab:jtweed|@jtweed]] was approved. * 10:24 [[gitlab:ifrahkh|@ifrahkh]] was approved. * 09:06 [[gitlab:wikibayer|@wikibayer]] was approved. === 2024-10-06 === * 10:27 [[gitlab:keerthan16|@keerthan16]] was approved. === 2024-10-04 === * 07:45 [[gitlab:hakimi97|@hakimi97]] was approved. === 2024-09-30 === * 07:39 [[gitlab:ninjastrikers|@ninjastrikers]] was approved. === 2024-09-28 === * 17:30 [[gitlab:webrunner95|@webrunner95]] was approved. === 2024-09-18 === * 21:39 [[gitlab:elliottetzkorn|@elliottetzkorn]] was approved. === 2024-09-14 === * 22:06 [[gitlab:humptydumpty|@humptydumpty]] was approved. === 2024-09-06 === * 08:48 [[gitlab:mickabarber|@mickabarber]] was approved. === 2024-08-27 === * 17:36 [[gitlab:edgars|@edgars]] was approved. === 2024-08-22 === * 09:18 [[gitlab:antonkokhwmde|@antonkokhwmde]] was approved. === 2024-08-14 === * 19:21 [[gitlab:jfk|@jfk]] was approved. === 2024-08-13 === * 17:57 [[gitlab:daxserver|@daxserver]] was approved. === 2024-08-11 === * 09:57 [[gitlab:pauliesnug|@pauliesnug]] was approved. === 2024-08-10 === * 08:42 [[gitlab:ashig|@ashig]] was approved. === 2024-08-09 === * 14:09 [[gitlab:masssly|@masssly]] was approved. === 2024-08-05 === * 22:15 [[gitlab:mrtortue|@mrtortue]] was approved. === 2024-08-02 === * 16:21 [[gitlab:dsantini|@dsantini]] was approved. === 2024-07-31 === * 11:54 [[gitlab:cptviraj|@cptviraj]] was approved. === 2024-07-30 === * 19:09 [[gitlab:iniquity|@iniquity]] was approved. * 10:00 [[gitlab:collins|@collins]] was approved. === 2024-07-27 === * 15:57 [[gitlab:songnguxyz|@songnguxyz]] was approved. === 2024-07-25 === * 12:36 [[gitlab:mszabo|@mszabo]] was approved. * 09:21 [[gitlab:agarwalmahima|@agarwalmahima]] was approved. === 2024-07-24 === * 08:05 [[gitlab:dragoniez|@dragoniez]] was approved. === 2024-07-23 === * 06:54 [[gitlab:mirji|@mirji]] was approved. === 2024-07-16 === * 10:00 [[gitlab:lakejason0|@lakejason0]] was approved. === 2024-07-12 === * 11:33 [[gitlab:cn|@cn]] was approved. * 08:12 [[gitlab:unchampignon|@unchampignon]] was approved. === 2024-07-07 === * 17:12 [[gitlab:agamyasamuel|@agamyasamuel]] was approved. * 05:24 [[gitlab:kuldeepburjbhalaike|@kuldeepburjbhalaike]] was approved. === 2024-07-06 === * 11:18 [[gitlab:dibya|@dibya]] was approved. * 04:54 [[gitlab:sarthakparashar|@sarthakparashar]] was approved. === 2024-07-05 === * 18:15 [[gitlab:vanshikarathi|@vanshikarathi]] was approved. === 2024-07-02 === * 19:00 [[gitlab:ebrahim|@ebrahim]] was approved. === 2024-07-01 === * 20:12 [[gitlab:rockingpenny4|@rockingpenny4]] was approved. * 18:15 [[gitlab:balajijagadesh|@balajijagadesh]] was approved. === 2024-06-30 === * 18:24 [[gitlab:hrideshmg|@hrideshmg]] was approved. * 07:18 [[gitlab:chanakyakumardas|@chanakyakumardas]] was approved. * 06:30 [[gitlab:rihaan180|@rihaan180]] was approved. === 2024-06-27 === * 17:36 [[gitlab:driedmueller|@driedmueller]] was approved. === 2024-06-19 === * 12:57 [[gitlab:audreypenven|@audreypenven]] was approved. === 2024-06-16 === * 01:18 [[gitlab:roysmith|@roysmith]] was approved. === 2024-06-08 === * 02:45 [[gitlab:jleedev|@jleedev]] was approved. === 2024-06-03 === * 13:57 [[gitlab:afeder|@afeder]] was approved. === 2024-06-01 === * 10:54 [[gitlab:florianschmitt|@florianschmitt]] was approved. === 2024-05-30 === * 16:42 [[gitlab:krlsca|@krlsca]] was approved. === 2024-05-28 === * 11:24 [[gitlab:rickijay|@rickijay]] was approved. === 2024-05-26 === * 11:18 [[gitlab:ranjithsiji|@ranjithsiji]] was approved. === 2024-05-25 === * 07:24 [[gitlab:jony|@jony]] was approved. === 2024-05-23 === * 08:45 [[gitlab:lepticed7|@lepticed7]] was approved. === 2024-05-22 === * 20:42 [[gitlab:echecs|@echecs]] was approved. === 2024-05-21 === * 13:33 [[gitlab:mbs|@mbs]] was approved. === 2024-05-19 === * 18:06 [[gitlab:ionenlaser|@ionenlaser]] was approved. === 2024-05-18 === * 23:36 [[gitlab:mdaniels5757|@mdaniels5757]] was approved. === 2024-05-17 === * 08:54 [[gitlab:grapedog|@grapedog]] was approved. === 2024-05-08 === * 19:42 [[gitlab:kelhurd|@kelhurd]] was approved. * 19:06 [[gitlab:khurd|@khurd]] was approved. === 2024-05-06 === * 19:48 [[gitlab:j3j5|@j3j5]] was approved. * 12:06 [[gitlab:tk-999|@tk-999]] was approved. === 2024-05-05 === * 22:09 [[gitlab:pppery|@pppery]] was approved. * 20:33 [[gitlab:sakretsu|@sakretsu]] was approved. * 12:12 [[gitlab:waterquark|@waterquark]] was approved. === 2024-05-04 === * 09:03 [[gitlab:multichill|@multichill]] was approved. * 07:42 [[gitlab:abaris|@abaris]] was approved. === 2024-05-03 === * 14:57 [[gitlab:maurusian|@maurusian]] was approved. === 2024-04-24 === * 05:48 [[gitlab:wolfinux|@wolfinux]] was approved. === 2024-04-23 === * 15:48 [[gitlab:dreamrimmer|@dreamrimmer]] was approved. === 2024-04-21 === * 06:51 [[gitlab:alon|@alon]] was approved. === 2024-04-17 === * 23:33 [[gitlab:derenrich|@derenrich]] was approved. === 2024-04-16 === * 17:18 [[gitlab:valcio|@valcio]] was approved. === 2024-04-14 === * 16:51 [[gitlab:wikilucas00|@wikilucas00]] was approved. === 2024-04-06 === * 12:48 [[gitlab:theprotonade|@theprotonade]] was approved. === 2024-04-02 === * 07:30 [[gitlab:bohuizhang|@bohuizhang]] was approved. === 2024-03-30 === * 13:36 [[gitlab:lpintscher|@lpintscher]] was approved. === 2024-03-26 === * 17:09 [[gitlab:eenabulele|@eenabulele]] was approved. === 2024-03-25 === * 14:27 [[gitlab:tuukka|@tuukka]] was approved. === 2024-03-24 === * 12:24 [[gitlab:firefly|@firefly]] was approved. === 2024-03-21 === * 19:33 [[gitlab:universal-omega|@universal-omega]] was approved. === 2024-03-17 === * 10:36 [[gitlab:bisel91|@bisel91]] was approved. === 2024-03-16 === * 10:09 [[gitlab:delord|@delord]] was approved. * 00:42 [[gitlab:athulvis1|@athulvis1]] was approved. === 2024-03-15 === * 19:06 [[gitlab:ignaciorodrguez|@ignaciorodrguez]] was approved. * 08:30 [[gitlab:peachey88|@peachey88]] was approved. * 06:51 [[gitlab:derick|@derick]] was approved. === 2024-03-12 === * 15:06 [[gitlab:xiaoxiao|@xiaoxiao]] was approved. === 2024-03-06 === * 13:21 [[gitlab:desianabae1|@desianabae1]] was approved. === 2024-03-05 === * 19:21 [[gitlab:ep1c|@ep1c]] was approved. * 16:33 [[gitlab:jasmine|@jasmine]] was approved. === 2024-03-02 === * 06:42 [[gitlab:potsdamlamb|@potsdamlamb]] was approved. === 2024-02-29 === * 23:18 [[gitlab:arandomname123|@arandomname123]] was approved. * 18:03 [[gitlab:baba|@baba]] was approved. * 17:48 [[gitlab:yfdyh000|@yfdyh000]] was approved. * 03:09 [[gitlab:sds|@sds]] was approved. === 2024-02-27 === * 23:33 [[gitlab:lofhi|@lofhi]] was approved. === 2024-02-15 === * 19:45 [[gitlab:gergesshamon|@gergesshamon]] was approved. === 2024-02-14 === * 14:33 [[gitlab:philipnelson99|@philipnelson99]] was approved. === 2024-02-13 === * 13:06 [[gitlab:dringsim|@dringsim]] was approved. === 2024-02-12 === * 17:36 [[gitlab:haak|@haak]] was approved. === 2024-02-05 === * 17:33 [[gitlab:qwerfjkl|@qwerfjkl]] was approved. * 17:14 [[gitlab:ahecht|@ahecht]] was approved. === 2024-02-01 === * 09:27 [[gitlab:arinaigum|@arinaigum]] was approved. * 00:15 [[gitlab:jas42|@jas42]] was approved. * 00:15 [[gitlab:edhu|@edhu]] was approved. * 00:15 [[gitlab:marnanel|@marnanel]] was approved. * 00:15 [[gitlab:ibrahemqasim|@ibrahemqasim]] was approved. * 00:15 [[gitlab:amasotti|@amasotti]] was approved. * 00:15 [[gitlab:deni|@deni]] was approved. * 00:15 [[gitlab:cyber|@cyber]] was approved. * 00:15 [[gitlab:saroj|@saroj]] was approved. === 2024-01-29 === * 21:42 [[gitlab:rgupta|@rgupta]] was approved. === 2024-01-07 === * 09:48 [[gitlab:lutrome|@lutrome]] was approved. === 2024-01-05 === * 20:48 [[gitlab:jinoytommanjaly|@jinoytommanjaly]] was approved. * 02:51 [[gitlab:braunobruno|@braunobruno]] was approved. * 01:08 [[gitlab:amorymeltzer|@amorymeltzer]] was approved. * 01:08 [[gitlab:phi22ipus|@phi22ipus]] was approved. === 2024-01-03 === * 14:45 [[gitlab:gabina|@gabina]] was approved. === 2024-01-02 === * 13:18 [[gitlab:arthurtaylor|@arthurtaylor]] was approved. === 2023-12-23 === * 00:33 [[gitlab:aram|@aram]] was approved. === 2023-12-22 === * 16:24 [[gitlab:elpitareio|@elpitareio]] was approved. === 2023-12-21 === * 00:43 [[gitlab:bsadowski1|@bsadowski1]] was approved. * 00:43 [[gitlab:ederporto|@ederporto]] was approved. * 00:43 [[gitlab:sadraiiali|@sadraiiali]] was approved. * 00:43 [[gitlab:wasp-outis|@wasp-outis]] was approved. * 00:43 [[gitlab:bodhisattwa|@bodhisattwa]] was approved. * 00:43 [[gitlab:air7538|@air7538]] was approved. * 00:43 [[gitlab:anzx|@anzx]] was approved. * 00:43 [[gitlab:tekask1903|@tekask1903]] was approved. * 00:42 [[gitlab:kiwi-0x010c|@kiwi-0x010c]] was approved. * 00:42 [[gitlab:mpaa|@mpaa]] was approved. * 00:42 [[gitlab:kutay|@kutay]] was approved. * 00:42 [[gitlab:wattmto|@wattmto]] was approved. 9advnqwrlr5ebpy3vqoofgm174u59vr 0 455357 2452808 2447752 2026-08-31T08:42:20Z BJensen-WMF 48942 /* Troubleshooting */ Add manual certificate renewal instructions. 2452808 wikitext text/x-wiki <!-- scrollbot-page: Mw-mcrouter --> {{Navigation Wikimedia infrastructure|expand=mw}} This is the daemonset proxying all mediawiki memcached requests, to our [[Memcached|memcached cluster]]. It is running our beloved [https://github.com/facebook/mcrouter mcrouter] For how MediaWiki itself uses the cache (WANObjectCache, ParserCache, MicroStash), see [[Memcached for MediaWiki]]. For the underlying clusters, pools and servers, see [[Memcached]]. [[File:Memcached-infra-1.jpg|alt=mcrouter memcached infra at WMF|thumb|466x466px|mcrouter memcached infra at WMF|none]] == mcrouter image and exporter == Image is in the [[gerrit:plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/mcrouter/|production images repo]], where the defaults are set. Image version in production is defined in the [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/deployment_server/kubernetes.yaml|puppet repo]] under <code>profile::kubernetes::deployment_server::general:common_images</code> == Daemonset == mw-mcrouter is running as a daemonset, i.e. every k8s node is running an instance of it. === Service === <code>mw-mcrouter</code> is using the mcrouter chart. Notable keys in <code>values.yaml</code>: * <code>cache:mcrouter:public_service: true</code> enables mcrouter as a standalone service * <code>service:use_node_local_endpoints: true</code> routes requests to the node-local endpoint of a pod * <code>cache:mcrouter:service:clusterIP</code>Static IP (per DC) where the service listens, as defined in [[Kubernetes/Service_ips]] ** eqiad ClusterIP: 10.64.72.12 ** codfw ClusterIP: 10.192.72.12 ===== values-eqiad.yaml ===== <syntaxhighlight lang="yaml">cache: mcrouter: service: clusterIP: 10.64.72.12 enabled: true route_prefix: eqiad/mw zone: eqiad routes: - route: /eqiad/mw pool: eqiad-servers failover_time: 600 - route: /codfw/mw pool: codfw-servers failover_time: 600 - route: /eqiad/mw-wan failover_time: 600 pool: eqiad-servers replica: route: /codfw/mw-wan pool: codfw-servers</syntaxhighlight> === Deployment === In the mcrouter chart, in daemonset.yaml, this ds is configured to update a few pods at a time. <syntaxhighlight lang="yaml"> spec: updateStrategy: type: RollingUpdate rollingUpdate: maxUnavailable: 4 </syntaxhighlight>During an mw-mcrouter deployment: * Generally, mcrouter's configuration or image are rarely in need for an update * In case of new image being pulled, deployment may take as long as 15' ** This is due to <code>maxUnavailable: 1</code> above * Alerts for elevated [https://logstash.wikimedia.org/app/dashboards#/view/memcached?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3h%2Cto%3Anow)) mw-memcached errors] * Alerts for mw-mcrouter helmfile being in a bad state * All the alerts above will clear * If deployment is stuck due to eg a node having insufficient resources to host the mcrouter === Testing changes === The safest way to test changes in mcrouter is to switch the mw-debug mediawiki deployment to use the in-pod mcrouter container. This is described in the next section === Switching Mediawiki to in-pod mcrouter container === While it sounds complicated, it is not. To switch <code>mw-debug</code> in eqiad to use the in-pod container, the following stanza must be added: ===== mw-debug/values-eqiad.yaml ===== <syntaxhighlight lang="yaml">cache: mcrouter: enabled: true route_prefix: eqiad/mw zone: eqiad routes: - route: /eqiad/mw pool: eqiad-servers failover_time: 600 - route: /codfw/mw pool: codfw-servers failover_time: 600 - route: /eqiad/mw-wan pool: eqiad-servers failover_time: 600 replica: route: /codfw/mw-wan pool: codfw-servers # Wikifunctions routes, omitted in production # - route: /local/wf # pool: wf-eqiad # # No failover for wikifunction # failover_time: 0 # # use only if testing new images # common_images: # mcrouter: # mcrouter: mcrouter:2023.07.17.00-1-20240714 # exporter: prometheus-mcrouter-exporter:0.0.1-3-20240714 php: envvars: MCROUTER_SERVER: "127.0.0.1:11213" STATSD_EXPORTER_PROMETHEUS_SERVICE_HOST: false # MCROUTER_SERVER: "10.64.72.12:4442" # mcrouter-main.mw-mcrouter.svc.cluster.local </syntaxhighlight> == Replacing a host in a pool == To replace a host in a mcrouter pool (see [[Memcached#Clusters and pools|Memcached clusters and pools]] and [[Memcached#Servers|Memcached servers]]): # Edit <code>profile::mediawiki::mcrouter_wancache::shards</code> in <code>hieradata/common/profile/mediawiki/mcrouter_wancache.yaml</code> in the operations/puppet repo with the new server, e.g. [https://gerrit.wikimedia.org/r/c/operations/puppet/+/1315946 this change]. # Merge and run puppet on the active deployment server. # Redeploy mw-mcrouter via helmfile. This is fast, since it is basically a reload of mcrouter's pool configuration. #* Other changes, e.g. adding <code>extra_args</code> (such as in <code>charts/mcrouter/.fixtures/mcrouter-command-line-options.yaml</code>), would instead trigger a pod restart. === Taking a host offline for maintenance === Whether a host should be removed from the shard list, or simply left in place and allowed to fail over, depends on which pool it belongs to: * '''Gutter pool''': if a Gutter server is going to be offline for maintenance or any other reason, it '''MUST''' be removed from the configuration. The Gutter pool is only a 3-shard-per-DC fallback (see [[Memcached#Clusters and pools|Memcached clusters and pools]]), so it has no further fallback of its own if one of its own shards is unavailable. * '''Main pool''': if a Main server is going to be offline, regardless of the expected duration, it is '''preferable to leave it in the shard list''' and let the [[Memcached#Clusters and pools|gutter pool]] pick up its traffic via the normal TKO/failover mechanism (see [[mw-mcrouter#Routes|Main route]]). The reason for this is that '''order matters''' in the mcrouter shard list. Removing a server from the middle of the list causes every server listed after it to be re-hashed to a different position, i.e. all the keys previously held by those later shards effectively become cache misses and need to be recomputed and re-stored. Simply leaving a down Main host in the list and letting Gutter handle its traffic avoids this mass re-hashing/recompute, at the cost of temporarily using the Gutter pool, which is fine really. == Routes == Each MediaWiki pod accesses memcached through its local Mcrouter instance. Mcrouter introduces the concepts of [https://github.com/facebook/mcrouter/wiki/Prefix-routing-setup routes] and pools and each route applies consistent hashing on the key name to know where to send it, i.e. which of the 18 shards for memcached (see [[Memcached#Servers|Memcached clusters and pools]]). There are several routes available in our configuration, which are addressable via a route prefix that mcrouter '''strips from the key''' before forwarding the memcached command. # '''Main route'''. This route is declared as <code>/$region/mw/</code> but is not addressed by MediaWiki as such. It routes to the dc-local "Main" pool shards. If a shard is perceived as unavailable from a pod ("TKO") the local mcrouter forwards all commands (incl gets, sets, and locks) to a shard of the "Gutter" pool instead. #* This route is used by the majority of traffic, through <code>WANObjectCache::getWithSet</code> calls in [[Memcached for MediaWiki#WANObjectCache|MediaWiki]]. #* MediaWiki doesn't use the <code>/$region/mw/</code> prefix. Instead <code>/$region/mw/</code> is the default route and MediaWiki sends these commands without any routing prefix. #* Switchover to and from the gutterpool is decided by Mcrouter locally. The keys stored in a gutter server have a reduced TTL. === Example === The memcached key <code>WANCache:v:metawiki:translate-groups</code> (belongs to the [[mw:Extension:Translate|Translate extension]]) is formatted by the WANCache library. When Translate wants to get the value of this key, WANCache will send a <code>GET</code> command from MediaWiki to <code>localhost:11213</code>, where mcrouter is listening. The command is then further routed to <code>mc1022</code> (based on key hashing). MediaWiki is totally ignorant about the <code>mc[1,2]0XX</code> host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where keys are hashed/routed to:<syntaxhighlight lang="bash"> elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2 VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16 10.64.0.83:11211 END elukey@mw1345:~$ dig -x 10.64.0.83 +short mc1022.eqiad.wmnet. </syntaxhighlight>Some things to notice: * The special prefix <code>__mcrouter__.route</code> is intercepted by mcrouter. These are admin commands, for which proxy returns directly without contacting the memcached hosts. This function returns the target of the consistent hashing of the key name. * Mcrouter listens on port 11213 on all MediaWiki [[Application servers|app servers]], meanwhile on every <code>mc10XX</code> host memcached listens on port 11211. To get a key and dump it to a file it is sufficient to:<syntaxhighlight lang="bash"> elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt elukey@mw1345:~$ du -hs dump.txt 380K dump.txt </syntaxhighlight>In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content. == Troubleshooting == When dealing with Kubernetes, your answer may be found in [[Kubernetes/Kubectl/Cheat_Sheet|Kubernetes kubectl Cheat Sheet]] === Memcached server is down === That is ok, the gutter pool will pick up its traffic. === Memcached Gutter Pool server is down, and we need the Gutter Pool === In this case: * [[Memcached#Servers|the gutter pool server in question MUST be removed from the configuration]] * Merge in puppet * Run puppet on the active deployment server * Deploy mw-mcrouter === Deployment is stalled === If you are deploying a new version of the daemonset, but you see pods stuck in the previous version, and elevate mw-memcached errors: * Check the daemonset's status <syntaxhighlight lang="text"> jiji@deploy1002:~$ kubectl get ds NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE mcrouter-main 210 210 210 210 210 <none> 30d </syntaxhighlight> * Check events and pods to find which node may be stalling the rollout ** If it is a resource problem (e.g. insufficient CPU), you may kill a random pod from the node in question (as root) <syntaxhighlight lang="text"> kube_env admin eqiad kubectl -n mw-mcrouter get events --sort-by=.metadata.creationTimestamp kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=wikikube-worker1001.eqiad.wmnet kubectl -n mw-api-ext delete po mw-api-ext.eqiad.main-koko-lala </syntaxhighlight> === Certificates failed to renew === PKI certificates ought to use their [[PKI/Clients#Puppet integration|Puppet integration]] to pull down renewed certificates automatically. In an emergency, it is possible to manually renew certificates using the process [[PKI/Clients#CLI Interactions|here]]. == Dashboards == * [https://grafana-rw.wikimedia.org/d/ltSHWhHIk/mw-mcrouter mw-mcrouter] * [https://logstash.wikimedia.org/app/dashboards#/view/memcached?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-3h%2Cto%3Anow)) mw-memcached errors] * [https://grafana-rw.wikimedia.org/d/000000316/memcache Memcached Dashboard] * [https://grafana-rw.wikimedia.org/d/000000316/memcache?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-cluster=memcached_gutter&var-instance=All Memcached Gutterpool status] == Links == * [[gerrit:plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/mcrouter/|mcrouter image]] * [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/deployment_server/kubernetes.yaml|mcrouter image versions]] * [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/mediawiki/mcrouter_wancache.yaml|list of mediawiki memcached production servers]] == See also == * [[Memcached]]: clusters, pools and servers proxied by mw-mcrouter. * [[Memcached for MediaWiki]]: how MediaWiki uses the cache (WANObjectCache, ParserCache, MicroStash). [[Category:How-To]] [[Category:Caching]] [[Category:MediaWiki production]] [[Category:SRE Service Operations]] 3oz6xi4c6d8gadorihj2nl0bgp13a8i 0 455928 2452785 2452747 2026-08-31T03:28:06Z CMing (WMF) 23277 /* Testing */ 2452785 wikitext text/x-wiki This page provides a guide to using the '''JavaScript Test Kitchen SDK''' for instrument and experiment creators. == Setup == To set up a local development environment for writing instrument code, follow the [[Test Kitchen/Local development setup|setup guide for MediaWiki and Test Kitchen]]. For more context and tooling recommendations, see the [[Test Kitchen/Maintainer introduction|introduction to Test Kitchen development]]. == Create an instrument == Assuming that you have registered already your instrument in [https://test-kitchen.wikimedia.org Test Kitchen UI], you can instantiate an instrument in your instrumentation code as follows: <syntaxhighlight lang="javascript"> const instrument = await mw.testKitchen.getInstrument( 'my-machine-readable-instrument-name' ); </syntaxhighlight> Once you've created an instrument, you can submit an event using <code>send</code> == Send a click event == <code>send</code> provides a simplified method to send any kind of event that use the Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics base schemas]. Here's an example of an instrument in JavaScript that uses <code>Instrument#send</code> to submit an event when a user clicks on an interwiki link. <syntaxhighlight lang="javascript"> function getLinkInteractionData( jqEvent ) { const link = jqEvent.target; return { action_source: link.href, action_context: link.title }; } const instrument = await mw.testKitchen.getInstrument( 'my-machine-readable-instrument-name' ); // 'a.extiw' will match anchors that have a extiw class. extiw is used for interwiki links. $( '#content' ).on( 'click', 'a.extiw', ( jqEvent ) => instrument.send( 'click', getLinkInteractionData( jqEvent ) ) );</syntaxhighlight> The resulting event: * includes <code>action: click</code> * includes optional interaction data (<code>action_source</code> and <code>action_context</code>) * is validated against the latest Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base base schema for web], as set in <code>newInstrument</code> * is published to the specified event stream (in this case, <code>mediawiki.product_metrics.example</code>), as it will have been defined when registering the instrument in Test Kitchen UI == Submit an interaction event == An interaction event is meant to represent a basic interaction with some target or some event occurring. For example, a user hovers over a UI element or an app notifies the server of its current state. Here's an example of an instrument in JavaScript that uses <code>Instrument#send</code> to send an event when a user hovers over an interwiki link. <syntaxhighlight lang="javascript">$( '#content' ).on( 'mouseover', 'a.extiw', ( jqEvent ) => instrument.send( 'hover', getLinkInteractionData( jqEvent ) ) );</syntaxhighlight> The resulting event: * includes the specified value of <code>action</code> * includes optional interaction data you have provided in <code>getLinkInteractionData</code> (<code>action_source</code> and <code>action_context</code>) * is validated against the specified schema (in this case, the Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base base schema for web]), as it will have been defined when registering the instrument in Test Kitchen UI * is published to the specified event stream (in this case, <code>mediawiki.product_metrics.example</code>), as it will have been defined when registering the instrument in Test Kitchen UI == Reusable instrumentation == Both <code>[https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#88 Experiment]</code> and <code>[https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#224 Instrument]</code> implement [https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#49 ExperimentInterface]. They send analytics events and differ only in the context attached to those events. === The <code>use()</code> pattern === The <code>use()</code> method was developed to enable reusable, event sender-agnostic instrumentation. It lets you attach generic, reusable instrumentation to an experiment or an instrument. The instrumentation is written once and then can be applied to either an experiment or an instrument without change. The instrumentation callback runs only when the user is enrolled in the experiment or is in sample for the instrument. <syntaxhighlight lang="javascript"> mw.testKitchen.getInstrument( 'my-instrument' ).use( myInstrumentation ); mw.testKitchen.getExperiment( 'my-experiment' ) .then( ( experiment ) => experiment.use( myInstrumentation ) ); </syntaxhighlight> === API === {| class="wikitable" |+ experiment.use( instrumentation ) / instrument.use( instrumentation ) |- | Parameter || instrumentation - a [https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#82 mw.testKitchen.GenericInstrumentation] callback |- | Returns || The same experiment/instrument (chainable) |} The callback is invoked synchronously with the experiment/instrument itself typed as an <code>EventSenderInterface</code>: <syntaxhighlight lang="javascript"> /** * @param {mw.testKitchen.EventSenderInterface} eventSender */ function myInstrumentation( eventSender ) { // Set up whatever you need, then send events through `eventSender`. eventSender.send( 'page-visited', interactionData, contextualAttributes ); } </syntaxhighlight> The only method generic instrumentation should depend on: {| class="wikitable" |+ <code>EventSenderInterface#send( action, interactionData, contextualAttributes )</code> |- ! Parameter !! Type !! Description |- | action || string || The action taken, e.g. 'click' |- | interactionData || Object (optional) || Additional data about the action |- | contextualAttributes || string[] (optional} || Per-event contextual attributes merged with those from config |- |} {{Note|type=reminder|text=When the sender is an experiment, the event is automatically decorated with experiment enrollment data before being sent.}} === How <code>use()</code> works === Depending on the sender's state, <code>use()</code> behaves differently: {| class="wikitable" |- ! Sender !! <code>use()</code> behavior |- | Enrolled <code>Experiment</code> || Runs the instrumentation, sends events |- | <code>UnenrolledExperiment</code> || No-op - the instrumentation is never called |- | In-sample <code>Instrument</code> || Runs the instrumentation, sends events |- | <code>UnsampledInstrument</code> || No-op - the instrumentation is never called |- | <code>OverriddenExperiment</code> || Runs the instrumentation, but <code>send()</code> logs to the console |} Because unenrolled/unsampled senders make <code>use()</code> a no-op, any setup inside your instrumentation (event listeners, mw.hook, DOM queries, etc) runs only for users who are actually in the experiment/sample. No need to check for enrollment group or if in sample. <syntaxhighlight lang="javascript"> mw.testKitchen.getInstrument( 'donate-ctr' ).use( ( eventSender ) => { mw.hook( 'donate.click' ).add( ( data ) => { eventSender.send( 'donate-btn-click', { action_context: data.target } ); } ); } ); </syntaxhighlight> === Reusable instrumentation module example === A generic instrumentation module is a just a function that takes an <code>EventSenderInterface</code>: <syntaxhighlight lang="javascript"> // clickThroughRateInstrument.js function clickThroughRateInstrument( eventSender ) { mw.hook( 'donate.click' ).add( ( data ) => { eventSender.send( 'donate-btn-click', { action_context: data.action_context } ); } ); } </syntaxhighlight> Apply it to an instrument: <syntaxhighlight lang="javascript"> const ctr = require( './clickThroughRateInstrument.js' ): mw.testKitchen.getInstrument( 'donate-instrument' ).use( ctr ); </syntaxhighlight> Apply it to an experiment: <syntaxhighlight lang="javascript"> const ctr = require( './clickThroughRateInstrument.js' ): mw.testKitchen.getExperiment( 'donate-experiment' ) .then( ( experiment ) => experiment.use( ctr ) ); </syntaxhighlight> {{Note|type=reminder|text=<code>mw.testKitchen.getExperiment()</code> is asynchronous and resolves with an <code>ExperimentInterface</code> while <code>mw.testKitchen.getInstrument()</code> is synchronous and returns an <code>InstrumentInterface</code> directly.}} === Chaining === <code>use()</code> returns the sender so calls can be chained. For example, apply generic instrumentation and then send an experiment exposure event: <syntaxhighlight lang="javascript"> mw.testKitchen.getExperiment( 'donate-experiment' ) .then( ( experiment ) => experiment.use( ctr ).sendExposure() ); </syntaxhighlight> === Testing === Because instrumentation only depends on <code>EventSenderInterface</code>, you can test it without the SDK by passing in a stub sender: <syntaxhighlight lang="javascript"> QUnit.test( 'CTR instrument sends a donate-btn-click event', ( assert ) => { const sent = []; const fakeSender = { send: ( action, data ) => sent.push( { action, ...data } ) }; clickThroughRateInstrument( fakeSender ); mw.hook ( 'donate.click' ).fire( { action_context: 'cta' } ); assert.deepEqual( sent, [ { action: 'donate-btn-click', action_context: 'cta' } ] ); } ); </syntaxhighlight> For integration testing that goes through <code>getExperiment()</code> or <code>getInstrument</code>, use <code>mw.testKitchen.useFakeExperiments()</code> or <code>mw.testKitchen.useFakeInstruments()</code> to stub enrollment or sampling. === <code>use()</code> vs. <code>send()</code> === <code>use()</code> is additive, it does not replace <code>send()</code>. Calling <code>send()</code> directly on an experiment or an instrument still works and is appropriate for one-off events. Consider <code>use()</code> when you have instrumentation you want to reuse across senders and/or run only when the user is enrolled in an experiment or in sample for an instrument. == Implementation == The JavaScript SDK is provided by [[mw:Extension:TestKitchen|the TestKitchen extension]]. The EventLogging extension maintains a list of streams to be included in the module, <code>$wgEventLoggingStreamNames</code>, which can be used to minimize the size of the module. When <code>$wgEventLoggingStreamNames</code> is falsy the JavaScript SDK will not validate whether the destination stream is configured before submitting the event to the destination event service. == Reference == For up-to-date documentation, see [[wmdoc:TestKitchen/master/js/|the JavaScript SDK documentation microsite]]. For parameter descriptions and validation rules, see the [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base web schema definition]. px3qk0dx3h8kr3kicm4rzhedhmy9jsr 2452787 2452785 2026-08-31T03:30:58Z CMing (WMF) 23277 /* Testing */ 2452787 wikitext text/x-wiki This page provides a guide to using the '''JavaScript Test Kitchen SDK''' for instrument and experiment creators. == Setup == To set up a local development environment for writing instrument code, follow the [[Test Kitchen/Local development setup|setup guide for MediaWiki and Test Kitchen]]. For more context and tooling recommendations, see the [[Test Kitchen/Maintainer introduction|introduction to Test Kitchen development]]. == Create an instrument == Assuming that you have registered already your instrument in [https://test-kitchen.wikimedia.org Test Kitchen UI], you can instantiate an instrument in your instrumentation code as follows: <syntaxhighlight lang="javascript"> const instrument = await mw.testKitchen.getInstrument( 'my-machine-readable-instrument-name' ); </syntaxhighlight> Once you've created an instrument, you can submit an event using <code>send</code> == Send a click event == <code>send</code> provides a simplified method to send any kind of event that use the Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics base schemas]. Here's an example of an instrument in JavaScript that uses <code>Instrument#send</code> to submit an event when a user clicks on an interwiki link. <syntaxhighlight lang="javascript"> function getLinkInteractionData( jqEvent ) { const link = jqEvent.target; return { action_source: link.href, action_context: link.title }; } const instrument = await mw.testKitchen.getInstrument( 'my-machine-readable-instrument-name' ); // 'a.extiw' will match anchors that have a extiw class. extiw is used for interwiki links. $( '#content' ).on( 'click', 'a.extiw', ( jqEvent ) => instrument.send( 'click', getLinkInteractionData( jqEvent ) ) );</syntaxhighlight> The resulting event: * includes <code>action: click</code> * includes optional interaction data (<code>action_source</code> and <code>action_context</code>) * is validated against the latest Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base base schema for web], as set in <code>newInstrument</code> * is published to the specified event stream (in this case, <code>mediawiki.product_metrics.example</code>), as it will have been defined when registering the instrument in Test Kitchen UI == Submit an interaction event == An interaction event is meant to represent a basic interaction with some target or some event occurring. For example, a user hovers over a UI element or an app notifies the server of its current state. Here's an example of an instrument in JavaScript that uses <code>Instrument#send</code> to send an event when a user hovers over an interwiki link. <syntaxhighlight lang="javascript">$( '#content' ).on( 'mouseover', 'a.extiw', ( jqEvent ) => instrument.send( 'hover', getLinkInteractionData( jqEvent ) ) );</syntaxhighlight> The resulting event: * includes the specified value of <code>action</code> * includes optional interaction data you have provided in <code>getLinkInteractionData</code> (<code>action_source</code> and <code>action_context</code>) * is validated against the specified schema (in this case, the Test Kitchen [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base base schema for web]), as it will have been defined when registering the instrument in Test Kitchen UI * is published to the specified event stream (in this case, <code>mediawiki.product_metrics.example</code>), as it will have been defined when registering the instrument in Test Kitchen UI == Reusable instrumentation == Both <code>[https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#88 Experiment]</code> and <code>[https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#224 Instrument]</code> implement [https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#49 ExperimentInterface]. They send analytics events and differ only in the context attached to those events. === The <code>use()</code> pattern === The <code>use()</code> method was developed to enable reusable, event sender-agnostic instrumentation. It lets you attach generic, reusable instrumentation to an experiment or an instrument. The instrumentation is written once and then can be applied to either an experiment or an instrument without change. The instrumentation callback runs only when the user is enrolled in the experiment or is in sample for the instrument. <syntaxhighlight lang="javascript"> mw.testKitchen.getInstrument( 'my-instrument' ).use( myInstrumentation ); mw.testKitchen.getExperiment( 'my-experiment' ) .then( ( experiment ) => experiment.use( myInstrumentation ) ); </syntaxhighlight> === API === {| class="wikitable" |+ experiment.use( instrumentation ) / instrument.use( instrumentation ) |- | Parameter || instrumentation - a [https://gerrit.wikimedia.org/g/mediawiki/extensions/TestKitchen/+/cb716f4d30d4cecd8bc6aa40f9efe0ff4318cd1c/modules/ext.testKitchen/types.js#82 mw.testKitchen.GenericInstrumentation] callback |- | Returns || The same experiment/instrument (chainable) |} The callback is invoked synchronously with the experiment/instrument itself typed as an <code>EventSenderInterface</code>: <syntaxhighlight lang="javascript"> /** * @param {mw.testKitchen.EventSenderInterface} eventSender */ function myInstrumentation( eventSender ) { // Set up whatever you need, then send events through `eventSender`. eventSender.send( 'page-visited', interactionData, contextualAttributes ); } </syntaxhighlight> The only method generic instrumentation should depend on: {| class="wikitable" |+ <code>EventSenderInterface#send( action, interactionData, contextualAttributes )</code> |- ! Parameter !! Type !! Description |- | action || string || The action taken, e.g. 'click' |- | interactionData || Object (optional) || Additional data about the action |- | contextualAttributes || string[] (optional} || Per-event contextual attributes merged with those from config |- |} {{Note|type=reminder|text=When the sender is an experiment, the event is automatically decorated with experiment enrollment data before being sent.}} === How <code>use()</code> works === Depending on the sender's state, <code>use()</code> behaves differently: {| class="wikitable" |- ! Sender !! <code>use()</code> behavior |- | Enrolled <code>Experiment</code> || Runs the instrumentation, sends events |- | <code>UnenrolledExperiment</code> || No-op - the instrumentation is never called |- | In-sample <code>Instrument</code> || Runs the instrumentation, sends events |- | <code>UnsampledInstrument</code> || No-op - the instrumentation is never called |- | <code>OverriddenExperiment</code> || Runs the instrumentation, but <code>send()</code> logs to the console |} Because unenrolled/unsampled senders make <code>use()</code> a no-op, any setup inside your instrumentation (event listeners, mw.hook, DOM queries, etc) runs only for users who are actually in the experiment/sample. No need to check for enrollment group or if in sample. <syntaxhighlight lang="javascript"> mw.testKitchen.getInstrument( 'donate-ctr' ).use( ( eventSender ) => { mw.hook( 'donate.click' ).add( ( data ) => { eventSender.send( 'donate-btn-click', { action_context: data.target } ); } ); } ); </syntaxhighlight> === Reusable instrumentation module example === A generic instrumentation module is a just a function that takes an <code>EventSenderInterface</code>: <syntaxhighlight lang="javascript"> // clickThroughRateInstrument.js function clickThroughRateInstrument( eventSender ) { mw.hook( 'donate.click' ).add( ( data ) => { eventSender.send( 'donate-btn-click', { action_context: data.action_context } ); } ); } </syntaxhighlight> Apply it to an instrument: <syntaxhighlight lang="javascript"> const ctr = require( './clickThroughRateInstrument.js' ): mw.testKitchen.getInstrument( 'donate-instrument' ).use( ctr ); </syntaxhighlight> Apply it to an experiment: <syntaxhighlight lang="javascript"> const ctr = require( './clickThroughRateInstrument.js' ): mw.testKitchen.getExperiment( 'donate-experiment' ) .then( ( experiment ) => experiment.use( ctr ) ); </syntaxhighlight> {{Note|type=reminder|text=<code>mw.testKitchen.getExperiment()</code> is asynchronous and resolves with an <code>ExperimentInterface</code> while <code>mw.testKitchen.getInstrument()</code> is synchronous and returns an <code>InstrumentInterface</code> directly.}} === Chaining === <code>use()</code> returns the sender so calls can be chained. For example, apply generic instrumentation and then send an experiment exposure event: <syntaxhighlight lang="javascript"> mw.testKitchen.getExperiment( 'donate-experiment' ) .then( ( experiment ) => experiment.use( ctr ).sendExposure() ); </syntaxhighlight> === Testing === Because instrumentation only depends on <code>EventSenderInterface</code>, you can test it without the SDK by passing in a stub sender: <syntaxhighlight lang="javascript"> QUnit.test( 'CTR instrument sends a donate-btn-click event', ( assert ) => { const sent = []; const fakeSender = { send: ( action, data ) => sent.push( { action, ...data } ) }; clickThroughRateInstrument( fakeSender ); mw.hook ( 'donate.click' ).fire( { action_context: 'homepage' } ); assert.deepEqual( sent, [ { action: 'donate-btn-click', action_context: 'homepage' } ] ); } ); </syntaxhighlight> For integration testing that goes through <code>getExperiment()</code> or <code>getInstrument</code>, use <code>mw.testKitchen.useFakeExperiments()</code> or <code>mw.testKitchen.useFakeInstruments()</code> to stub enrollment or sampling. === <code>use()</code> vs. <code>send()</code> === <code>use()</code> is additive, it does not replace <code>send()</code>. Calling <code>send()</code> directly on an experiment or an instrument still works and is appropriate for one-off events. Consider <code>use()</code> when you have instrumentation you want to reuse across senders and/or run only when the user is enrolled in an experiment or in sample for an instrument. == Implementation == The JavaScript SDK is provided by [[mw:Extension:TestKitchen|the TestKitchen extension]]. The EventLogging extension maintains a list of streams to be included in the module, <code>$wgEventLoggingStreamNames</code>, which can be used to minimize the size of the module. When <code>$wgEventLoggingStreamNames</code> is falsy the JavaScript SDK will not validate whether the destination stream is configured before submitting the event to the destination event service. == Reference == For up-to-date documentation, see [[wmdoc:TestKitchen/master/js/|the JavaScript SDK documentation microsite]]. For parameter descriptions and validation rules, see the [https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/product_metrics/web/base web schema definition]. n7vszw9p383pli81gighy5wbjd64ycq 0 456526 2452878 2449640 2026-08-31T11:43:25Z CGoubert-WMF 32775 /* How do I restart all of my team's failed jobs? */ 2452878 wikitext text/x-wiki {{lowercase title}} {{see|'''Cron''' redirects here. You might be looking for [[Help:Toolforge/Running jobs#Creating scheduled jobs (cron jobs)]].}} {{Note|type=reminder|text=This page documents the new Kubernetes way of running scheduled MediaWiki Maintenance scripts. If you discover issues with a migrated Maintenance script, please report this via to the relevant of {{phab|T341555}}.}} {{Navigation Wikimedia infrastructure|expand=mw}} '''mw-cron''' is a [[MediaWiki On Kubernetes]] deployment in [[WikiKube]], for automatically executing MediaWiki maintenance scripts on a regular schedule. SRE is targetting March 2025 to complete migration from the [[Maintenance server]] to the <code>mw-cron</code> deployment. == Dashboards == * [https://w.wiki/DocP Logstash mw-cron warnings and errors] * [https://grafana.wikimedia.org/d/f8cb4b3b-8db9-4446-afa7-c293839964b8/jobs-cronjobs-mediawiki-mw-cron Grafana mw-cron job statuses and resource information] == Monitoring == === Alerting === By default, we are setting up two alerts: # A warning for <code>team=serviceops</code> that will show up in [https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=team%3Dserviceops ServiceOp's AlertManager dashboard] # Either a [[Alertmanager#How_do_I_get_an_alert_to_open_a_task?|Phabricator task]] on the owning team's PHID, or a slack message on their channel, depending on preference Please note that in the case of a Phabricator alert, it will open a task titled <code>MediaWiki periodic job $cronjob_name failed</code> and, as long as the task is open, future alerts firing for that team and job will update the description. === Probe === For now, the probe fires when a <code>Job</code> has failed, no matter the reason (eviction, non-0 exit code, etc.). It will not autoresolve on a subsequent successful run, and requires manual deletion of the failed <code>Job</code> to resolve. == Creating a new periodic job == Given that deployment-prep still uses systemd timers for periodic jobs, some parameters are duplicated due to incompatibilities between crontab and systemd timer syntax. In puppet, create your periodic_job resource:<syntaxhighlight lang="puppet">profile::mediawiki::periodic_job { 'myteam-periodicjob': command => '/usr/local/bin/foreachwikiindblist /srv/mediawiki/dblists/large.dblist extensions/blah/maintenance/periodic_job.php', interval => '*-*-* *:11:00', # systemd-timer syntax, for deployment-prep cron_schedule => '11 * * * *', # crontab syntax, for mw-cron on kubernetes kubernetes => true, # In production, create the resource on kubernetes, not the maintenance server team => 'myteam', # AlertManager team tag, should lead to a proper alert route in modules/alertmanager/templates/alertmanager.yml.erb script_label => 'periodic_job.php-large', # Your script name, potentially augmented of the wiki or section description => 'Run blah-ish periodic job on large wikis', concurrency_policy => 'Forbid', # This can be ommited if your job can be restarted if it overruns, the default is Replace. startingdeadlineseconds => 1800, # This should be ommited if you do not specify concurrency_policy: Forbid ttlsecondsafterfinished => 604800, # Default in helm chart 604800 (1 week), how long should finished jobs stay after completion. Should be minimum 2*interval to have some history. helmfile_defaults_dir => $helmfile_defaults_dir, # Mandatory }</syntaxhighlight>Then include it in [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/kubernetes/deployment_server/mediawiki/periodic_jobs.pp|modules/profile/manifests/kubernetes/deployment_server/mediawiki/periodic_jobs.pp]] (for kubernetes) and [[gerrit:plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/mediawiki/maintenance.pp|modules/profile/manifests/mediawiki/maintenance.pp]] (for bare metal/beta). Then after a proper review and PCC check, you can move on to [[Mw-cron jobs#Deploying periodic jobs|Deploying periodic jobs]]. === What's the deal with concurrency_policy and startingdeadlineseconds? === Jobs that need to complete and not be restarted if they run longer than the time between schedules should set <code>concurrency_policy: Forbid</code> and a proper (around half the interval, but can be fine tuned) <code>startingDeadlineSeconds</code> Jobs that can handle or need to restart should stick to the default settings (<code>concurrency_policy: Replace</code>). They will get killed and restarted on schedule. More information in [[phab:T394423|⚓ T394423 Investigate startingDeadlineSeconds setting for kubernetes CronJobs]] == Deploying periodic jobs == # Merge your puppet change # Run puppet on the currently active deployment server<syntaxhighlight lang="shell-session"> deploy1003:~$ sudo run-puppet-agent </syntaxhighlight> # Deploy the changes to <code>mw-cron</code> using <code>helmfile</code> on the currently active datacentre<syntaxhighlight lang="shell-session"> deploy1003:~$ cd /srv/deployment-charts/helmfile.d/services/mw-cron deploy1003:~$ helmfile -e eqiad -i apply --context 5 </syntaxhighlight> # The job will start on its next schedule, but you can use [[#Manually running a CronJob]] to execute it ahead of schedule. == Troubleshooting == If your job has failed, you can either look at logstash, or diagnose from the command line on the deployment server. === Logs from kubernetes === If one of your maintenance scripts recently failed, a Phabricator task will have been opened for your team ([https://phabricator.wikimedia.org/T391574 example]). The task description provides a more specific <code>kubectl get jobs</code> command invocation that lists only failed jobs specific to the maintenance script that prompted the task. The example below assumes Eqiad is the primary datacenter. 1. Enter the kubectl scope for the mw-cron cluster in the primary datacenter. <syntaxhighlight lang="shell-session"> deploy1003:~$ kube-env mw-cron eqiad # If that fails with a permission denied error, you may be a member of the restricted group. In that case, use: deploy1003:~$ kube-env mw-cron-restricted eqiad </syntaxhighlight> 2. list recently failed jobs, or, list all jobs<syntaxhighlight lang="shell-session"> deploy1003:~$ kubectl get jobs --field-selector status.successful=0 deploy1003:~$ kubectl get jobs NAMESPACE NAME COMPLETIONS DURATION AGE mw-cron example-something-29050030 0/1 9s 28m </syntaxhighlight> 3. access the logs from the pod (insert job name after "jobs/", and select mediawiki container)<syntaxhighlight lang="shell-session"> deploy1003:~$ kubectl logs jobs/example-something-29050030 mediawiki-main-app Doing stuff for things... ...found this. ...did that. Done! </syntaxhighlight> === Manually deleting a failed Job === In order to reset alerting, the failed job(s) need to be manually deleted. <syntaxhighlight lang="shell-session">deploy1003:~$ kube-env mw-cron-deploy eqiad deploy1003:~$ kubectl get jobs NAMESPACE NAME COMPLETIONS DURATION AGE mw-cron example-something-29050030 0/1 9s 28m deploy1003:~$ kubectl delete job example-something-29050030</syntaxhighlight> Or alternatively, if you are a member of the "restricted" group: <syntaxhighlight lang="shell-session">deploy1003:~$ kube-env mw-cron-restricted-deploy eqiad deploy1003:~$ kubectl get jobs NAMESPACE NAME COMPLETIONS DURATION AGE mw-cron example-something-29050030 0/1 9s 28m deploy1003:~$ kubectl delete job example-something-29050030</syntaxhighlight> === Manually running a CronJob === <syntaxhighlight lang="shell-session">deploy1003:~$ kube-env mw-cron-deploy eqiad deploy1003:~$ kubectl create job mediawiki-main-serviceops-version-$(date +"%Y%m%d%H%M") --from=cronjobs/mediawiki-main-serviceops-version job.batch/mediawiki-main-serviceops-version-202504081112 created deploy1003:$ kubectl get jobs -l 'team=sre-serviceops, cronjob=serviceops-version' NAME COMPLETIONS DURATION AGE mediawiki-main-serviceops-version-202504081112 1/1 49s 81s [...]</syntaxhighlight>It's a good idea to delete the job once your manual run is done and you don't need information from it anymore<syntaxhighlight lang="shell-session">deploy1003:~$ kubectl delete job mediawiki-main-serviceops-version-202504081112</syntaxhighlight> === Why is my job stopping after one error? === The default behaviour is to <code>set +e</code> and exit on the first error encountered. If your job is running on multiple wikis using <code>foreachwiki</code> or equivalent, and this is not desirable, you can set <code>foreachwiki_ignore_errors => true</code> in the puppet definition of your job. === My job doesn't do any calls to other services, can I avoid erroring out on service mesh unavailability? === You can add <code>mesh_check_skip => true,</code> to your <code>profile::mediawiki::periodic_job</code> definition and it will skip the availability check for the mesh. Additional configuration options will be provided to retry the complete run on this error when [[phab:T390972|T390972]] is resolved. === How do I restart all of my team's failed jobs? === Be careful as this will launch them all at the same time, if you have a lot of jobs or they can't run concurrently you may have a problem.<syntaxhighlight lang="shell-session"> deploy2003:~$ kube-env mw-cron-deploy eqiad deploy2003:~$ team=myteam # This will display: # - The failed job in the first column # - The cronjob it gets created from in the second # - The number of failed jobs in the third deploy2003:~$ kubectl get jobs -l team=${team} -o custom-columns=NAME:.metadata.name,CRONJOB:.metadata.ownerReferences[0].name,FAILED:.status.failed --no-headers | awk '$3==1 {print}' # You can then loop on the second column to get a display of all jobs for these cronjobs # This allows you to see if there have been successful jobs after the failure # This example uses `team=growth` because of what I had on hand deploy2003:~$ kubectl get jobs -l team=${team} -o custom-columns=NAME:.metadata.name,CRONJOB:.metadata.ownerReferences[0].name,FAILED:.status.failed --no-headers | awk '$3==1 {print $2}' | while read cj; do kubectl get jobs -l cronjob=${cj}; done NAME STATUS COMPLETIONS DURATION AGE growthexperiments-listtaskcounts-29784431 Failed 0/1 20h 20h growthexperiments-listtaskcounts-29785451 Complete 1/1 51m 3h growthexperiments-listtaskcounts-29785511 Complete 1/1 52m 120m growthexperiments-listtaskcounts-29785571 Complete 1/1 49m 60m growthexperiments-listtaskcounts-29785631 Running 0/1 57s 57s NAME STATUS COMPLETIONS DURATION AGE growthexperiments-updatementeedata-s2-29784615 Failed 0/1 16h 16h growthexperiments-updatementeedata-s2-29785155 Complete 1/1 10m 7h56m growthexperiments-updatementeedata-s2-29785335 Complete 1/1 10m 4h56m growthexperiments-updatementeedata-s2-29785515 Complete 1/1 10m 116m # Here, since all jobs have a more recent run that succeeded, we can just go ahead and delete all the failed jobs to reset alerting deploy2003:~$ kubectl get jobs -l team=${team} -o custom-columns=NAME:.metadata.name,CRONJOB:.metadata.ownerReferences[0].name,FAILED:.status.failed --no-headers | awk '$3==1 {print $1}' | xargs kubectl delete job # If we wanted to reschedule all of them (truncates job name to 63 chars otherwise kubernetes fails) deploy2003:~$ kubectl get jobs -l team=${team} -o custom-columns=NAME:.metadata.name,CRONJOB:.metadata.ownerReferences[0].name,FAILED:.status.failed --no-headers | awk '$3==1 {print $2}' | while read cj; do job=${cj}-manual-$(date +"%Y%m%d%H%M"); kubectl create job ${job:0:63} --from=cronjobs/${cj}; done </syntaxhighlight> == Job Migration == ===General procedure=== ====Code changes==== The jobs are still defined in puppet, using the <code>profile::mediawiki::periodic_job</code> resource. Additional parameters are necessary to migrate a job to <code>mw-cron</code> and remove it from the maintenance servers. * If the periodic jobs are defined in a subprofile of <code>profile:mediawiki::maintenance</code>, change the class definition to include the <code>$helmfile_defaults_dir</code> parameter <syntaxhighlight lang=puppet> class profile::mediawiki::maintenance::subprofile( Stdlib::Unixpath $helmfile_defaults_dir = lookup('profile::kubernetes::deployment_server::global_config::general_dir', {default_value => '/etc/helmfile-defaults'}), ) { </syntaxhighlight> * Include your subprofile in <code>profile::kubernetes::deployment_server::mediawiki::periodic_jobs</code> * Add the following additional parameters: <syntaxhighlight lang=puppet> cron_schedule => '*/10 * * * *', # The interval must be converted from systemd-calendar intervals to crontab syntax. Keep the interval parameter as well if your job is used on beta kubernetes => true, # Create the CronJob resource in mw-cron, and remove the systemd-timer from the maintenance server team => 'job-owner-team', # For easier monitoring, log dashboard, and alerting script_label => 'scriptName-wikiName', # A label for monitoring, logging, and alerting, preferably the script name and its target description => 'A longer form description of the periodic job', helmfile_defaults_dir => $helmfile_defaults_dir, # Pass down the directory where the jobs will be defined </syntaxhighlight> ====Example==== We recommend the following procedure: * '''[Done in a batch, not necessary anymore]''' Create a first change adding the subprofile to the <code>profile::kubernetes::deployment_server::mediawiki::periodic_jobs</code> profile, like {{Gerrit|1117234}}. This will have all the no-op changes. * Migrate the jobs in follow-up patches like {{Gerrit|1117862}} ====Deployment==== # Disable puppet on the maintenance server if the job can't be interrupted easily # Merge the puppet change # Run puppet on the deployment server, this will create the job definition, but it won't be deployed to <code>mw-cron</code> yet # Stop the job on the maintenance server if it is running (use <code>systemctl status mediawiki_job_<jobname>.service</code> to check, <code>systemctl stop mediawiki_job_<jobname>.service</code> to stop). If the script is currently running, check with the responsible team if it's ok to stop it. # Deploy the <code>mw-cron</code> change with helmfile on the deployment server. The job will start on its next scheduled trigger. # Enable puppet on the maintenance server and run it. This will delete the systemd timer for the maintenance job. # If needed, use [[Mw-cron jobs#Manually running a CronJob|Manually_running_a_CronJob]] to trigger a manual run. [[Category:MediaWiki production]] ivxed7gwi4p8hgkzuksotny0jumavr1 116 460070 2452775 2436836 2026-08-30T14:38:50Z EtherpadBackupBot 54504 EYG02 archived by Ericliu1912 2452775 wikitext text/x-wiki === 2026-08-30 === * 14:38 [[etherpad:p/EYG02]] was archived as [[etherpadbackup:EYG02]] by [[User:Ericliu1912|Ericliu1912]] === 2026-07-15 === * 10:59 [[etherpad:p/mathml-mediawiki-2025]] was archived as [[etherpadbackup:mathml-mediawiki-2025]] by [[User:Krinkle|Krinkle]] === 2026-07-11 === * 21:27 [[etherpad:p/Future_of_Wikimedia_2026-07-07]] was archived as [[etherpadbackup:Future_of_Wikimedia_2026-07-07]] by [[User:Pppery|Pppery]] * 14:02 [[etherpad:p/WikimediaEvents]] was archived as [[etherpadbackup:WikimediaEvents]] by [[User:Exec8|Exec8]] === 2026-07-06 === * 21:30 [[etherpad:p/MWStake]] was archived as [[etherpadbackup:MWStake]] by [[User:Pppery|Pppery]] === 2026-06-21 === * 15:11 [[etherpad:p/IN-KL_Meetup_April_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_April_2026]] by [[User:Gnoeee|Gnoeee]] * 15:11 [[etherpad:p/IN-KL_Meetup_May_2026]] was archived as [[etherpadbackup:IN-KL_Meetup_May_2026]] by [[User:Gnoeee|Gnoeee]] === 2026-06-15 === * 09:55 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Ericliu1912|Ericliu1912]] === 2026-06-13 === * 12:08 [[etherpad:p/EYG01]] was archived as [[etherpadbackup:EYG01]] by [[User:Chlod|Chlod]] === 2026-06-08 === * 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]] * 09:23 [[etherpad:p/umlsisXFA4ql4IF1gi2-]] was archived as [[etherpadbackup:umlsisXFA4ql4IF1gi2-]] by [[User:Aristorkle|Aristorkle]] === 2026-05-29 === * 03:47 [[etherpad:p/ESEAPCon26_sessions_csv]] was archived as [[etherpadbackup:ESEAPCon26_sessions_csv]] by [[User:Robertsky|Robertsky]] * 03:41 [[etherpad:p/ESEAPCon26_day2_Room2_DB9UHD]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_DB9UHD]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_JD3H99]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_JD3H99]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Outofvenue_DBKL7W]] was archived as [[etherpadbackup:ESEAPCon26_day3_Outofvenue_DBKL7W]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_3KZYPS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_3KZYPS]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_YF7VEZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_YF7VEZ]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_8TXHAF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_8TXHAF]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_QRT8BV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_QRT8BV]] by [[User:Robertsky|Robertsky]] * 03:36 [[etherpad:p/ESEAPCon26_day3_Room3_RFAU8M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room3_RFAU8M]] by [[User:Robertsky|Robertsky]] * 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_VU3QL8]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_VU3QL8]] by [[User:Robertsky|Robertsky]] * 03:35 [[etherpad:p/ESEAPCon26_day3_Room2_M3QUZD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_M3QUZD]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_TFFBBH]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_TFFBBH]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_PQYYHB]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_PQYYHB]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_CA7G3X]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_CA7G3X]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room2_8RX8NZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room2_8RX8NZ]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_JYNFKE]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_JYNFKE]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_LFV39K]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_LFV39K]] by [[User:Robertsky|Robertsky]] * 03:34 [[etherpad:p/ESEAPCon26_day3_Room1_H9JCWN]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_H9JCWN]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_9EF8DZ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9EF8DZ]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_8EQWQL]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_8EQWQL]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_A87WBV]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_A87WBV]] by [[User:Robertsky|Robertsky]] * 03:33 [[etherpad:p/ESEAPCon26_day3_Room1_3K3T9M]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_3K3T9M]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_9DJJP7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_9DJJP7]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_AKYUKC]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_AKYUKC]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Room1_S99FWD]] was archived as [[etherpadbackup:ESEAPCon26_day3_Room1_S99FWD]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_HGHY9E]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_HGHY9E]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_YL8ST7]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_YL8ST7]] by [[User:Robertsky|Robertsky]] * 03:32 [[etherpad:p/ESEAPCon26_day3_Mainroom_8ZNVWS]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_8ZNVWS]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_C7ZRET]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_C7ZRET]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_NCHDPG]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_NCHDPG]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_TW9WFJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_TW9WFJ]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_VNX3UJ]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_VNX3UJ]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_BQHSPF]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_BQHSPF]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day3_Mainroom_ZNUEAR]] was archived as [[etherpadbackup:ESEAPCon26_day3_Mainroom_ZNUEAR]] by [[User:Robertsky|Robertsky]] * 03:30 [[etherpad:p/ESEAPCon26_day2_Outofvenue_VVR7KR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_VVR7KR]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Outofvenue_CJTRQM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Outofvenue_CJTRQM]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_DEXKVE]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DEXKVE]] by [[User:Robertsky|Robertsky]] * 03:29 [[etherpad:p/ESEAPCon26_day2_Room3_7DSNAX]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7DSNAX]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_DZGGRB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_DZGGRB]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_7MBXLG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_7MBXLG]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_Q3XQYZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_Q3XQYZ]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_TDXWE3]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_TDXWE3]] by [[User:Robertsky|Robertsky]] * 03:28 [[etherpad:p/ESEAPCon26_day2_Room3_HZZZKL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_HZZZKL]] by [[User:Robertsky|Robertsky]] * 03:27 [[etherpad:p/ESEAPCon26_day2_Room2_JTBXUG]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JTBXUG]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_ALULGS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_ALULGS]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_MNBFWL]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_MNBFWL]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_RRHPKR]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_RRHPKR]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_AURUVZ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_AURUVZ]] by [[User:Robertsky|Robertsky]] * 03:26 [[etherpad:p/ESEAPCon26_day2_Room2_3MAKHH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_3MAKHH]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_TXFRZB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_TXFRZB]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_QUFR97]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QUFR97]] by [[User:Robertsky|Robertsky]] * 03:25 [[etherpad:p/ESEAPCon26_day2_Room2_JT7TFH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_JT7TFH]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_LJ3BL8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_LJ3BL8]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_9YDQCT]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_9YDQCT]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_GKN9F7]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_GKN9F7]] by [[User:Robertsky|Robertsky]] * 03:24 [[etherpad:p/ESEAPCon26_day2_Room1_CFUMZ9]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CFUMZ9]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_CRTN93]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_CRTN93]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_MGZMP8]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_MGZMP8]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Room1_8WQJKB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room1_8WQJKB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LLPZSB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LLPZSB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_HMSJPM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_HMSJPM]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_SWCDHM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_SWCDHM]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_3CDL7X]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_3CDL7X]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_AHCNHB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_AHCNHB]] by [[User:Robertsky|Robertsky]] * 03:22 [[etherpad:p/ESEAPCon26_day2_Mainroom_LTSVEH]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LTSVEH]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_ZVYUWS]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_ZVYUWS]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_T8YSUB]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_T8YSUB]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_LMXU7L]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_LMXU7L]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_YKMM9C]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_YKMM9C]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day2_Mainroom_RZKSWJ]] was archived as [[etherpadbackup:ESEAPCon26_day2_Mainroom_RZKSWJ]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Outofvenue_3RMDEL]] was archived as [[etherpadbackup:ESEAPCon26_day1_Outofvenue_3RMDEL]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room3_XAJUWY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room3_XAJUWY]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_FSZJK7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FSZJK7]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_W3ND3N]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_W3ND3N]] by [[User:Robertsky|Robertsky]] * 03:17 [[etherpad:p/ESEAPCon26_day1_Room2_8AR9U9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_8AR9U9]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_NUWQDM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_NUWQDM]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_DGZT88]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_DGZT88]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room2_FPWESV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room2_FPWESV]] by [[User:Robertsky|Robertsky]] * 03:15 [[etherpad:p/ESEAPCon26_day1_Room1_7EKDB9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_7EKDB9]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_XVFLQU]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_XVFLQU]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_KCR737]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_KCR737]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_EBATCZ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_EBATCZ]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_MGKRNF]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_MGKRNF]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Room1_DHFDW7]] was archived as [[etherpadbackup:ESEAPCon26_day1_Room1_DHFDW7]] by [[User:Robertsky|Robertsky]] * 03:14 [[etherpad:p/ESEAPCon26_day1_Mainroom_CLMDUY]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_CLMDUY]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_B7Y3PM]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_B7Y3PM]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_WGT9AQ]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_WGT9AQ]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_AXG3KB]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_AXG3KB]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_L8GRH9]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_L8GRH9]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_UAF8RA]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_UAF8RA]] by [[User:Robertsky|Robertsky]] * 03:11 [[etherpad:p/ESEAPCon26_day1_Mainroom_NTXNZV]] was archived as [[etherpadbackup:ESEAPCon26_day1_Mainroom_NTXNZV]] by [[User:Robertsky|Robertsky]] === 2026-05-26 === * 21:47 [[etherpad:p/Wikimedia_Hackathon_2026_Closing_Showcase]] was archived as [[etherpadbackup:Wikimedia_Hackathon_2026_Closing_Showcase]] by [[User:Gnoeee|Gnoeee]] * 21:47 [[etherpad:p/South_Asia_Community_Call]] was archived as [[etherpadbackup:South_Asia_Community_Call]] by [[User:Gnoeee|Gnoeee]] === 2026-05-23 === * 09:18 [[etherpad:p/ESEAPCon26_day2_Room3_G9S7TM]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room3_G9S7TM]] by [[User:Ericliu1912|Ericliu1912]] * 09:18 [[etherpad:p/ESEAPCon26_day2_Room2_QP3MFV]] was archived as [[etherpadbackup:ESEAPCon26_day2_Room2_QP3MFV]] by [[User:Ericliu1912|Ericliu1912]] * 02:46 [[etherpad:p/org_structure_reading_group]] was archived as [[etherpadbackup:org_structure_reading_group]] by [[User:BryanDavis|BryanDavis]] === 2026-05-18 === * 19:20 [[etherpad:p/Chartres]] was archived as [[etherpadbackup:Chartres]] by [[User:Sj|Sj]] * 19:08 [[etherpad:p/city-homelessness-okn]] was archived as [[etherpadbackup:city-homelessness-okn]] by [[User:Sj|Sj]] * 19:07 [[etherpad:p/WM-structural-problems-overview]] was archived as [[etherpadbackup:WM-structural-problems-overview]] by [[User:Sj|Sj]] * 19:03 [[etherpad:p/WLM-2021-May]] was archived as [[etherpadbackup:WLM-2021-May]] by [[User:Sj|Sj]] * 19:02 [[etherpad:p/WLM-2021-April]] was archived as [[etherpadbackup:WLM-2021-April]] by [[User:Sj|Sj]] * 19:02 [[etherpad:p/shexcg_minutes]] was archived as [[etherpadbackup:shexcg_minutes]] by [[User:Sj|Sj]] * 19:01 [[etherpad:p/ctp-letter]] was archived as [[etherpadbackup:ctp-letter]] by [[User:Sj|Sj]] * 18:59 [[etherpad:p/Wikimania_oral_history]] was archived as [[etherpadbackup:Wikimania_oral_history]] by [[User:Sj|Sj]] * 18:55 [[etherpad:p/covid19]] was archived as [[etherpadbackup:covid19]] by [[User:Sj|Sj]] * 18:54 [[etherpad:p/Covering_George_Floyd]] was archived as [[etherpadbackup:Covering_George_Floyd]] by [[User:Sj|Sj]] * 17:37 [[etherpad:p/JROSTS18-HACKDAY]] was archived as [[etherpadbackup:JROSTS18-HACKDAY]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFF]] was archived as [[etherpadbackup:JROSTS18-UNCFF]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFE]] was archived as [[etherpadbackup:JROSTS18-UNCFE]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFD]] was archived as [[etherpadbackup:JROSTS18-UNCFD]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFC]] was archived as [[etherpadbackup:JROSTS18-UNCFC]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCFB]] was archived as [[etherpadbackup:JROSTS18-UNCFB]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-OA.2]] was archived as [[etherpadbackup:JROSTS18-OA.2]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-UNCONFERENCING]] was archived as [[etherpadbackup:JROSTS18-UNCONFERENCING]] by [[User:Sj|Sj]] * 17:36 [[etherpad:p/JROSTS18-JROST]] was archived as [[etherpadbackup:JROSTS18-JROST]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-OB.1]] was archived as [[etherpadbackup:JROSTS18-OB.1]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-EOA]] was archived as [[etherpadbackup:JROSTS18-EOA]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-WFC.3]] was archived as [[etherpadbackup:JROSTS18-WFC.3]] by [[User:Sj|Sj]] * 17:35 [[etherpad:p/JROSTS18-WFC.2]] was archived as [[etherpadbackup:JROSTS18-WFC.2]] by [[User:Sj|Sj]] * 17:34 [[etherpad:p/JROSTS18-WFA.2]] was archived as [[etherpadbackup:JROSTS18-WFA.2]] by [[User:Sj|Sj]] * 17:34 [[etherpad:p/JROSTS18-WFA.1]] was archived as [[etherpadbackup:JROSTS18-WFA.1]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROSTS18-CROSSCUTTING]] was archived as [[etherpadbackup:JROSTS18-CROSSCUTTING]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROST-Master]] was archived as [[etherpadbackup:JROST-Master]] by [[User:Sj|Sj]] * 17:33 [[etherpad:p/JROSTS18-WFC.1]] was archived as [[etherpadbackup:JROSTS18-WFC.1]] by [[User:Sj|Sj]] * 17:19 [[etherpad:p/15.320]] was archived as [[etherpadbackup:15.320]] by [[User:Sj|Sj]] * 17:17 [[etherpad:p/medical_hotspots]] was archived as [[etherpadbackup:medical_hotspots]] by [[User:Sj|Sj]] * 17:16 [[etherpad:p/community_health]] was archived as [[etherpadbackup:community_health]] by [[User:Sj|Sj]] * 17:14 [[etherpad:p/whatsnext]] was archived as [[etherpadbackup:whatsnext]] by [[User:Sj|Sj]] * 17:12 [[etherpad:p/MWF20160318]] was archived as [[etherpadbackup:MWF20160318]] by [[User:Sj|Sj]] * 17:12 [[etherpad:p/MWF20160226]] was archived as [[etherpadbackup:MWF20160226]] by [[User:Sj|Sj]] * 17:11 [[etherpad:p/ceem14-pr]] was archived as [[etherpadbackup:ceem14-pr]] by [[User:Sj|Sj]] * 16:54 [[etherpad:p/Templates,_Page_Components_and_editing]] was archived as [[etherpadbackup:Templates,_Page_Components_and_editing]] by [[User:Sj|Sj]] * 16:54 [[etherpad:p/GLAM-Wikipedia-fr]] was archived as [[etherpadbackup:GLAM-Wikipedia-fr]] by [[User:Sj|Sj]] * 16:49 [[etherpad:p/editcontentmodel]] was archived as [[etherpadbackup:editcontentmodel]] by [[User:Sj|Sj]] * 16:43 [[etherpad:p/multimedia-weekly-meeting-2014-05-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-05-14]] by [[User:Sj|Sj]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-04]] was archived as [[etherpadbackup:cuny-wikiwed-2026-04]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-03]] was archived as [[etherpadbackup:cuny-wikiwed-2026-03]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2026-01]] was archived as [[etherpadbackup:cuny-wikiwed-2026-01]] by [[User:Pharos|Pharos]] * 16:06 [[etherpad:p/cuny-wikiwed-2025-11]] was archived as [[etherpadbackup:cuny-wikiwed-2025-11]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-10]] was archived as [[etherpadbackup:cuny-wikiwed-2025-10]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-09]] was archived as [[etherpadbackup:cuny-wikiwed-2025-09]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-08]] was archived as [[etherpadbackup:cuny-wikiwed-2025-08]] by [[User:Pharos|Pharos]] * 16:05 [[etherpad:p/cuny-wikiwed-2025-07]] was archived as [[etherpadbackup:cuny-wikiwed-2025-07]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-06]] was archived as [[etherpadbackup:cuny-wikiwed-2025-06]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-05]] was archived as [[etherpadbackup:cuny-wikiwed-2025-05]] by [[User:Pharos|Pharos]] * 16:04 [[etherpad:p/cuny-wikiwed-2025-04]] was archived as [[etherpadbackup:cuny-wikiwed-2025-04]] by [[User:Pharos|Pharos]] === 2026-05-10 === * 18:10 [[etherpad:p/wmhackshowcase2020]] was archived as [[etherpadbackup:wmhackshowcase2020]] by [[User:BryanDavis|BryanDavis]] * 18:03 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 18:01 [[etherpad:p/wdqs-500s]] was archived as [[etherpadbackup:wdqs-500s]] by [[User:BryanDavis|BryanDavis]] * 18:00 [[etherpad:p/wep_data_task_force_meeting_#1]] was archived as [[etherpadbackup:wep_data_task_force_meeting_#1]] by [[User:BryanDavis|BryanDavis]] === 2026-05-07 === * 00:35 [[etherpad:p/zh-ve-screenshots]] was archived as [[etherpadbackup:zh-ve-screenshots]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/zerocrash]] was archived as [[etherpadbackup:zerocrash]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/zero-mf-sync-up]] was archived as [[etherpadbackup:zero-mf-sync-up]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/yuvidump]] was archived as [[etherpadbackup:yuvidump]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/yuvi-knowledge-transfer]] was archived as [[etherpadbackup:yuvi-knowledge-transfer]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/yeZfmYkMu2]] was archived as [[etherpadbackup:yeZfmYkMu2]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/yAqOuwkDKuhwhwjvd44j]] was archived as [[etherpadbackup:yAqOuwkDKuhwhwjvd44j]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/y4gqz6<fe46Q]] was archived as [[etherpadbackup:y4gqz6<fe46Q]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/y4gqz6<fe46Q]] was archived as [[etherpadbackup:y4gqz6<fe46Q]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/xZLrPEASWV]] was archived as [[etherpadbackup:xZLrPEASWV]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/x-analytics]] was archived as [[etherpadbackup:x-analytics]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/wtf-is-production]] was archived as [[etherpadbackup:wtf-is-production]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/wtf]] was archived as [[etherpadbackup:wtf]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/wp-app-1-1-release-post]] was archived as [[etherpadbackup:wp-app-1-1-release-post]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/wmhack2021-phab]] was archived as [[etherpadbackup:wmhack2021-phab]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/wmhack2017-ciw]] was archived as [[etherpadbackup:wmhack2017-ciw]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/wmhack15-wikidata-architecture-session]] was archived as [[etherpadbackup:wmhack15-wikidata-architecture-session]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/wmh2024-leaving-coordination]] was archived as [[etherpadbackup:wmh2024-leaving-coordination]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/wmh2023-Toolhub,_Toolhunt,_and_the_quest]] was archived as [[etherpadbackup:wmh2023-Toolhub,_Toolhunt,_and_the_quest]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/wmh2023-Past,_Present_and_Future_of_Wikimedia]] was archived as [[etherpadbackup:wmh2023-Past,_Present_and_Future_of_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/wmh2023-Overview_of_technical_areas_%26_projects]] was archived as [[etherpadbackup:wmh2023-Overview_of_technical_areas_%26_projects]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] was archived as [[etherpadbackup:wmh2023-LLMs,_ChatGPT,_machine_learning_tools,_etc]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/wmh2023-Best_practices_&_ideas_for_implementing]] was archived as [[etherpadbackup:wmh2023-Best_practices_&_ideas_for_implementing]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/wmf11and12]] was archived as [[etherpadbackup:wmf11and12]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/wmcslightning]] was archived as [[etherpadbackup:wmcslightning]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/wmcs_nfs_future]] was archived as [[etherpadbackup:wmcs_nfs_future]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/wmcs-vs-rowb-upgrade]] was archived as [[etherpadbackup:wmcs-vs-rowb-upgrade]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/wmcs-goals-q4-16-17]] was archived as [[etherpadbackup:wmcs-goals-q4-16-17]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/wmcs-goals-q2-17-18]] was archived as [[etherpadbackup:wmcs-goals-q2-17-18]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/wmcs-fallout]] was archived as [[etherpadbackup:wmcs-fallout]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/wmcs-cumin-fail]] was archived as [[etherpadbackup:wmcs-cumin-fail]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/wmcs-contributors]] was archived as [[etherpadbackup:wmcs-contributors]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/wmcs-2023-06-07]] was archived as [[etherpadbackup:wmcs-2023-06-07]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/wmconf2013-evaluating_programs]] was archived as [[etherpadbackup:wmconf2013-evaluating_programs]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/wmconf2013-evaluating-programs]] was archived as [[etherpadbackup:wmconf2013-evaluating-programs]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/wlmapp-iteration1-story-estimation-and-review]] was archived as [[etherpadbackup:wlmapp-iteration1-story-estimation-and-review]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/wlm-sprint5-retro]] was archived as [[etherpadbackup:wlm-sprint5-retro]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/wlm-sprint4-retro]] was archived as [[etherpadbackup:wlm-sprint4-retro]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/wlm-sprint3-retro]] was archived as [[etherpadbackup:wlm-sprint3-retro]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/wlm-sprint2-retro]] was archived as [[etherpadbackup:wlm-sprint2-retro]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/wlm-sprint1-retro]] was archived as [[etherpadbackup:wlm-sprint1-retro]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/wlm-project-retro]] was archived as [[etherpadbackup:wlm-project-retro]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/wlm-on-labs-steps]] was archived as [[etherpadbackup:wlm-on-labs-steps]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/wlm-api-admin-tree]] was archived as [[etherpadbackup:wlm-api-admin-tree]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/wishathon-march-2024-wishlist-overview-session]] was archived as [[etherpadbackup:wishathon-march-2024-wishlist-overview-session]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/wishathon-march-2024-showcase]] was archived as [[etherpadbackup:wishathon-march-2024-showcase]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/wishathon-march-2024-office-hour]] was archived as [[etherpadbackup:wishathon-march-2024-office-hour]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/wishathon-march-2024-fotw-session]] was archived as [[etherpadbackup:wishathon-march-2024-fotw-session]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/wikitechrebuild]] was archived as [[etherpadbackup:wikitechrebuild]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/wikitech-extensions]] was archived as [[etherpadbackup:wikitech-extensions]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/wikireplica-dns-switch]] was archived as [[etherpadbackup:wikireplica-dns-switch]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/wikireplica-beta]] was archived as [[etherpadbackup:wikireplica-beta]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/wikipedia-mobile-1-2-beta1-notes]] was archived as [[etherpadbackup:wikipedia-mobile-1-2-beta1-notes]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/wikipedia-ios-4.0.5-release-notes]] was archived as [[etherpadbackup:wikipedia-ios-4.0.5-release-notes]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/wikipedia-app-saved-pages-architecture]] was archived as [[etherpadbackup:wikipedia-app-saved-pages-architecture]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/wikipedia-app-ios-issues]] was archived as [[etherpadbackup:wikipedia-app-ios-issues]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/wikimania_knowledge]] was archived as [[etherpadbackup:wikimania_knowledge]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/wikigroknextsteps]] was archived as [[etherpadbackup:wikigroknextsteps]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/wikigrokcollections]] was archived as [[etherpadbackup:wikigrokcollections]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/wikidev17-devwishlist]] was archived as [[etherpadbackup:wikidev17-devwishlist]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/wikidata_query_service_checkin]] was archived as [[etherpadbackup:wikidata_query_service_checkin]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/wikibits-undefault]] was archived as [[etherpadbackup:wikibits-undefault]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/wiki_technology_notes]] was archived as [[etherpadbackup:wiki_technology_notes]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/widgets-for-uploadwizard]] was archived as [[etherpadbackup:widgets-for-uploadwizard]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/whm2023-Overview_of_technical_areas_&_projects]] was archived as [[etherpadbackup:whm2023-Overview_of_technical_areas_&_projects]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/whats-left-in-toolforge-eqiad]] was archived as [[etherpadbackup:whats-left-in-toolforge-eqiad]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/what_is_left_in_tampa]] was archived as [[etherpadbackup:what_is_left_in_tampa]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/webservice-k8s-announce]] was archived as [[etherpadbackup:webservice-k8s-announce]] by [[User:BryanDavis|BryanDavis]] === 2026-05-06 === * 23:59 [[etherpad:p/webrequest-2021-Q4-loss]] was archived as [[etherpadbackup:webrequest-2021-Q4-loss]] by [[User:BryanDavis|BryanDavis]] * 23:59 [[etherpad:p/webreq_update]] was archived as [[etherpadbackup:webreq_update]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/we-really-love-nfs]] was archived as [[etherpadbackup:we-really-love-nfs]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/wdqs_graph_split_production_deploy]] was archived as [[etherpadbackup:wdqs_graph_split_production_deploy]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/wdqs-split-graph-rollout-plan]] was archived as [[etherpadbackup:wdqs-split-graph-rollout-plan]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/wdqs-2024-08-08]] was archived as [[etherpadbackup:wdqs-2024-08-08]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/wdio]] was archived as [[etherpadbackup:wdio]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/wcna_tools]] was archived as [[etherpadbackup:wcna_tools]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/wbq_test]] was archived as [[etherpadbackup:wbq_test]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/watchlist_id]] was archived as [[etherpadbackup:watchlist_id]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/wallaby-dev-magnum-install]] was archived as [[etherpadbackup:wallaby-dev-magnum-install]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/volans-tmp]] was archived as [[etherpadbackup:volans-tmp]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/vk-jumbo-cleanup]] was archived as [[etherpadbackup:vk-jumbo-cleanup]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/virt1006migrate]] was archived as [[etherpadbackup:virt1006migrate]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/vikassy]] was archived as [[etherpadbackup:vikassy]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/vikas_ve]] was archived as [[etherpadbackup:vikas_ve]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/vikas_vagrant]] was archived as [[etherpadbackup:vikas_vagrant]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/vikas-upload-error]] was archived as [[etherpadbackup:vikas-upload-error]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/vikas-ruby-style]] was archived as [[etherpadbackup:vikas-ruby-style]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/vikas-gerrit]] was archived as [[etherpadbackup:vikas-gerrit]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/vikas]] was archived as [[etherpadbackup:vikas]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/ve-future-thoughts-2015]] was archived as [[etherpadbackup:ve-future-thoughts-2015]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/ve-2016-05-05]] was archived as [[etherpadbackup:ve-2016-05-05]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/ve]] was archived as [[etherpadbackup:ve]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/varnishkafka_troubleshooting]] was archived as [[etherpadbackup:varnishkafka_troubleshooting]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/vandrew]] was archived as [[etherpadbackup:vandrew]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/uw-cucumber-sketch]] was archived as [[etherpadbackup:uw-cucumber-sketch]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/uw-bugday-2014-09-09]] was archived as [[etherpadbackup:uw-bugday-2014-09-09]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/uploadwizard-refactor-planning]] was archived as [[etherpadbackup:uploadwizard-refactor-planning]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/uploadwizard-browser-tests]] was archived as [[etherpadbackup:uploadwizard-browser-tests]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/uploadcampaigns-squids]] was archived as [[etherpadbackup:uploadcampaigns-squids]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/upload-tool-class-structure]] was archived as [[etherpadbackup:upload-tool-class-structure]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/unsophabricated]] was archived as [[etherpadbackup:unsophabricated]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/unittestingbrownbag]] was archived as [[etherpadbackup:unittestingbrownbag]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/unified]] was archived as [[etherpadbackup:unified]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/unaliased]] was archived as [[etherpadbackup:unaliased]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/unable_to_move_translated_page]] was archived as [[etherpadbackup:unable_to_move_translated_page]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/twocolconflictdeploy]] was archived as [[etherpadbackup:twocolconflictdeploy]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/twiterbot-debuging]] was archived as [[etherpadbackup:twiterbot-debuging]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/trusty]] was archived as [[etherpadbackup:trusty]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/trixie-announcement]] was archived as [[etherpadbackup:trixie-announcement]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/trend-prod]] was archived as [[etherpadbackup:trend-prod]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/trebuchet-propose]] was archived as [[etherpadbackup:trebuchet-propose]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/toomanythings]] was archived as [[etherpadbackup:toomanythings]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/toolssupport]] was archived as [[etherpadbackup:toolssupport]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/toolsdb-upgrade]] was archived as [[etherpadbackup:toolsdb-upgrade]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/toolsdb-10.6]] was archived as [[etherpadbackup:toolsdb-10.6]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/tools-trusty-move]] was archived as [[etherpadbackup:tools-trusty-move]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/tools-the-great-recompress]] was archived as [[etherpadbackup:tools-the-great-recompress]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/tools-reboots-cve-0728]] was archived as [[etherpadbackup:tools-reboots-cve-0728]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/tools-reboots]] was archived as [[etherpadbackup:tools-reboots]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/tools-puppet-prefixes]] was archived as [[etherpadbackup:tools-puppet-prefixes]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/tools-migration-nodes]] was archived as [[etherpadbackup:tools-migration-nodes]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/tools-migration]] was archived as [[etherpadbackup:tools-migration]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/tools-kubernetes-scenarios]] was archived as [[etherpadbackup:tools-kubernetes-scenarios]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/tools-k8s-chat-01]] was archived as [[etherpadbackup:tools-k8s-chat-01]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/toollabs20160127]] was archived as [[etherpadbackup:toollabs20160127]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/toollabs-webservice-trusty-switch]] was archived as [[etherpadbackup:toollabs-webservice-trusty-switch]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/toollabs-uwsgi]] was archived as [[etherpadbackup:toollabs-uwsgi]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/toollabs-quarter-goal-q4-2014-15]] was archived as [[etherpadbackup:toollabs-quarter-goal-q4-2014-15]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/toollabs-impact]] was archived as [[etherpadbackup:toollabs-impact]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/toollabs-ideal-getting-started]] was archived as [[etherpadbackup:toollabs-ideal-getting-started]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/toollabs-cdnjs]] was archived as [[etherpadbackup:toollabs-cdnjs]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/toolforge-reboots]] was archived as [[etherpadbackup:toolforge-reboots]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/toolforge-php72-webservice]] was archived as [[etherpadbackup:toolforge-php72-webservice]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/toolforge-k8s-upgrade-1.23]] was archived as [[etherpadbackup:toolforge-k8s-upgrade-1.23]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/toolforge-k8s-1.31]] was archived as [[etherpadbackup:toolforge-k8s-1.31]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/toolforge-k8s-1.28-to-1.29-upgrade]] was archived as [[etherpadbackup:toolforge-k8s-1.28-to-1.29-upgrade]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/toolforge-2021-12-14]] was archived as [[etherpadbackup:toolforge-2021-12-14]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/tool_labs-doc]] was archived as [[etherpadbackup:tool_labs-doc]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/tonythomas01]] was archived as [[etherpadbackup:tonythomas01]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/tofu]] was archived as [[etherpadbackup:tofu]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/todays-regressions]] was archived as [[etherpadbackup:todays-regressions]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/title_value]] was archived as [[etherpadbackup:title_value]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/things_to_restart_after_rabbit_explosion]] was archived as [[etherpadbackup:things_to_restart_after_rabbit_explosion]] by [[User:BryanDavis|BryanDavis]] * 06:03 [[etherpad:p/test]] was archived as [[etherpadbackup:test]] by [[User:BryanDavis|BryanDavis]] * 06:03 [[etherpad:p/termdata]] was archived as [[etherpadbackup:termdata]] by [[User:BryanDavis|BryanDavis]] * 06:02 [[etherpad:p/temp]] was archived as [[etherpadbackup:temp]] by [[User:BryanDavis|BryanDavis]] * 06:02 [[etherpad:p/teach-yuvi-to-walk]] was archived as [[etherpadbackup:teach-yuvi-to-walk]] by [[User:BryanDavis|BryanDavis]] * 06:01 [[etherpad:p/tcinterviewquestions]] was archived as [[etherpadbackup:tcinterviewquestions]] by [[User:BryanDavis|BryanDavis]] * 06:00 [[etherpad:p/tallinntransportation]] was archived as [[etherpadbackup:tallinntransportation]] by [[User:BryanDavis|BryanDavis]] * 06:00 [[etherpad:p/tabsinpuppet]] was archived as [[etherpadbackup:tabsinpuppet]] by [[User:BryanDavis|BryanDavis]] * 05:59 [[etherpad:p/tFyLAurf0Z]] was archived as [[etherpadbackup:tFyLAurf0Z]] by [[User:BryanDavis|BryanDavis]] * 05:59 [[etherpad:p/switchover]] was archived as [[etherpadbackup:switchover]] by [[User:BryanDavis|BryanDavis]] * 05:58 [[etherpad:p/swiftfail]] was archived as [[etherpadbackup:swiftfail]] by [[User:BryanDavis|BryanDavis]] * 05:58 [[etherpad:p/swhYPuS04T]] was archived as [[etherpadbackup:swhYPuS04T]] by [[User:BryanDavis|BryanDavis]] * 05:57 [[etherpad:p/supersetdeprecation]] was archived as [[etherpadbackup:supersetdeprecation]] by [[User:BryanDavis|BryanDavis]] * 05:57 [[etherpad:p/sulf]] was archived as [[etherpadbackup:sulf]] by [[User:BryanDavis|BryanDavis]] * 05:56 [[etherpad:p/styleguide]] was archived as [[etherpadbackup:styleguide]] by [[User:BryanDavis|BryanDavis]] * 05:56 [[etherpad:p/stuffs]] was archived as [[etherpadbackup:stuffs]] by [[User:BryanDavis|BryanDavis]] * 05:55 [[etherpad:p/stubsExpansion]] was archived as [[etherpadbackup:stubsExpansion]] by [[User:BryanDavis|BryanDavis]] * 05:55 [[etherpad:p/striker-prod-deploy]] was archived as [[etherpadbackup:striker-prod-deploy]] by [[User:BryanDavis|BryanDavis]] * 05:54 [[etherpad:p/striker-gitlab-20220906]] was archived as [[etherpadbackup:striker-gitlab-20220906]] by [[User:BryanDavis|BryanDavis]] * 05:54 [[etherpad:p/striker]] was archived as [[etherpadbackup:striker]] by [[User:BryanDavis|BryanDavis]] * 05:53 [[etherpad:p/streaming_updater_cutover]] was archived as [[etherpadbackup:streaming_updater_cutover]] by [[User:BryanDavis|BryanDavis]] * 05:53 [[etherpad:p/streaming-wdqs]] was archived as [[etherpadbackup:streaming-wdqs]] by [[User:BryanDavis|BryanDavis]] * 05:52 [[etherpad:p/stillnopms]] was archived as [[etherpadbackup:stillnopms]] by [[User:BryanDavis|BryanDavis]] * 05:52 [[etherpad:p/state_of_union_-_if_-network_bullet_points]] was archived as [[etherpadbackup:state_of_union_-_if_-network_bullet_points]] by [[User:BryanDavis|BryanDavis]] * 05:51 [[etherpad:p/stat1_accounts]] was archived as [[etherpadbackup:stat1_accounts]] by [[User:BryanDavis|BryanDavis]] * 05:51 [[etherpad:p/stat-analytics-vlan]] was archived as [[etherpadbackup:stat-analytics-vlan]] by [[User:BryanDavis|BryanDavis]] * 05:50 [[etherpad:p/standalone_masters]] was archived as [[etherpadbackup:standalone_masters]] by [[User:BryanDavis|BryanDavis]] * 05:50 [[etherpad:p/ssl]] was archived as [[etherpadbackup:ssl]] by [[User:BryanDavis|BryanDavis]] * 05:49 [[etherpad:p/ssh-key-change-trixie]] was archived as [[etherpadbackup:ssh-key-change-trixie]] by [[User:BryanDavis|BryanDavis]] * 05:48 [[etherpad:p/spark_pair_coding]] was archived as [[etherpadbackup:spark_pair_coding]] by [[User:BryanDavis|BryanDavis]] * 05:48 [[etherpad:p/spark2-planning]] was archived as [[etherpadbackup:spark2-planning]] by [[User:BryanDavis|BryanDavis]] * 05:47 [[etherpad:p/so1920.cs.unibo.it]] was archived as [[etherpadbackup:so1920.cs.unibo.it]] by [[User:BryanDavis|BryanDavis]] * 05:47 [[etherpad:p/sistersearch-AB-test]] was archived as [[etherpadbackup:sistersearch-AB-test]] by [[User:BryanDavis|BryanDavis]] * 05:46 [[etherpad:p/single-edit-tab]] was archived as [[etherpadbackup:single-edit-tab]] by [[User:BryanDavis|BryanDavis]] * 05:46 [[etherpad:p/shit_tim_says]] was archived as [[etherpadbackup:shit_tim_says]] by [[User:BryanDavis|BryanDavis]] * 05:45 [[etherpad:p/setting_up_ORES_lab_cluster]] was archived as [[etherpadbackup:setting_up_ORES_lab_cluster]] by [[User:BryanDavis|BryanDavis]] * 05:45 [[etherpad:p/sendBeacon-2014-10-03]] was archived as [[etherpadbackup:sendBeacon-2014-10-03]] by [[User:BryanDavis|BryanDavis]] * 05:44 [[etherpad:p/selenium]] was archived as [[etherpadbackup:selenium]] by [[User:BryanDavis|BryanDavis]] * 05:44 [[etherpad:p/security_group_tests]] was archived as [[etherpadbackup:security_group_tests]] by [[User:BryanDavis|BryanDavis]] * 05:43 [[etherpad:p/search_enhancements]] was archived as [[etherpadbackup:search_enhancements]] by [[User:BryanDavis|BryanDavis]] * 05:43 [[etherpad:p/search-logging]] was archived as [[etherpadbackup:search-logging]] by [[User:BryanDavis|BryanDavis]] * 05:42 [[etherpad:p/search-hypotesis-24-25]] was archived as [[etherpadbackup:search-hypotesis-24-25]] by [[User:BryanDavis|BryanDavis]] * 05:42 [[etherpad:p/search-503s]] was archived as [[etherpadbackup:search-503s]] by [[User:BryanDavis|BryanDavis]] * 05:41 [[etherpad:p/scrum-of-scrums]] was archived as [[etherpadbackup:scrum-of-scrums]] by [[User:BryanDavis|BryanDavis]] * 05:41 [[etherpad:p/scoping-troubles]] was archived as [[etherpadbackup:scoping-troubles]] by [[User:BryanDavis|BryanDavis]] * 05:40 [[etherpad:p/scap-commands]] was archived as [[etherpadbackup:scap-commands]] by [[User:BryanDavis|BryanDavis]] * 05:40 [[etherpad:p/scap-201617-q2]] was archived as [[etherpadbackup:scap-201617-q2]] by [[User:BryanDavis|BryanDavis]] * 05:39 [[etherpad:p/rubocop-2015-02-10]] was archived as [[etherpadbackup:rubocop-2015-02-10]] by [[User:BryanDavis|BryanDavis]] * 05:39 [[etherpad:p/rsyncnova]] was archived as [[etherpadbackup:rsyncnova]] by [[User:BryanDavis|BryanDavis]] * 05:38 [[etherpad:p/rollout-ldap-block-change]] was archived as [[etherpadbackup:rollout-ldap-block-change]] by [[User:BryanDavis|BryanDavis]] * 05:38 [[etherpad:p/role-mediainfo]] was archived as [[etherpadbackup:role-mediainfo]] by [[User:BryanDavis|BryanDavis]] * 05:37 [[etherpad:p/rockyupgrade]] was archived as [[etherpadbackup:rockyupgrade]] by [[User:BryanDavis|BryanDavis]] * 05:37 [[etherpad:p/robots.txt]] was archived as [[etherpadbackup:robots.txt]] by [[User:BryanDavis|BryanDavis]] * 05:36 [[etherpad:p/rmrebootsforcephmons]] was archived as [[etherpadbackup:rmrebootsforcephmons]] by [[User:BryanDavis|BryanDavis]] * 05:35 [[etherpad:p/revscoring_user_oriented]] was archived as [[etherpadbackup:revscoring_user_oriented]] by [[User:BryanDavis|BryanDavis]] * 05:35 [[etherpad:p/revscoring_process]] was archived as [[etherpadbackup:revscoring_process]] by [[User:BryanDavis|BryanDavis]] * 05:34 [[etherpad:p/revscoring_languages]] was archived as [[etherpadbackup:revscoring_languages]] by [[User:BryanDavis|BryanDavis]] * 05:34 [[etherpad:p/revscoring_hackathon]] was archived as [[etherpadbackup:revscoring_hackathon]] by [[User:BryanDavis|BryanDavis]] * 05:33 [[etherpad:p/review-workflow-yuvi]] was archived as [[etherpadbackup:review-workflow-yuvi]] by [[User:BryanDavis|BryanDavis]] * 05:33 [[etherpad:p/resurrect_omega]] was archived as [[etherpadbackup:resurrect_omega]] by [[User:BryanDavis|BryanDavis]] * 05:32 [[etherpad:p/resthack]] was archived as [[etherpadbackup:resthack]] by [[User:BryanDavis|BryanDavis]] * 05:32 [[etherpad:p/restbase-parsoid-storage-rollout]] was archived as [[etherpadbackup:restbase-parsoid-storage-rollout]] by [[User:BryanDavis|BryanDavis]] * 05:31 [[etherpad:p/researchers_in_our_midst]] was archived as [[etherpadbackup:researchers_in_our_midst]] by [[User:BryanDavis|BryanDavis]] * 05:31 [[etherpad:p/research_showcase_dec14]] was archived as [[etherpadbackup:research_showcase_dec14]] by [[User:BryanDavis|BryanDavis]] * 05:30 [[etherpad:p/replicas-temp-accounts]] was archived as [[etherpadbackup:replicas-temp-accounts]] by [[User:BryanDavis|BryanDavis]] * 05:30 [[etherpad:p/rep-lag-s8]] was archived as [[etherpadbackup:rep-lag-s8]] by [[User:BryanDavis|BryanDavis]] * 05:29 [[etherpad:p/remaining-ldap]] was archived as [[etherpadbackup:remaining-ldap]] by [[User:BryanDavis|BryanDavis]] * 05:29 [[etherpad:p/refinery_deploy]] was archived as [[etherpadbackup:refinery_deploy]] by [[User:BryanDavis|BryanDavis]] * 05:28 [[etherpad:p/red-links]] was archived as [[etherpadbackup:red-links]] by [[User:BryanDavis|BryanDavis]] * 05:28 [[etherpad:p/realm_networks]] was archived as [[etherpadbackup:realm_networks]] by [[User:BryanDavis|BryanDavis]] * 05:27 [[etherpad:p/reading-infra-2015-16-q4]] was archived as [[etherpadbackup:reading-infra-2015-16-q4]] by [[User:BryanDavis|BryanDavis]] * 05:27 [[etherpad:p/rdf-flink-k8s]] was archived as [[etherpadbackup:rdf-flink-k8s]] by [[User:BryanDavis|BryanDavis]] * 05:26 [[etherpad:p/rdbmess-2021]] was archived as [[etherpadbackup:rdbmess-2021]] by [[User:BryanDavis|BryanDavis]] * 05:26 [[etherpad:p/rcstream-misc]] was archived as [[etherpadbackup:rcstream-misc]] by [[User:BryanDavis|BryanDavis]] * 05:25 [[etherpad:p/rc1-release-email]] was archived as [[etherpadbackup:rc1-release-email]] by [[User:BryanDavis|BryanDavis]] * 05:25 [[etherpad:p/rc.2-announce]] was archived as [[etherpadbackup:rc.2-announce]] by [[User:BryanDavis|BryanDavis]] * 05:24 [[etherpad:p/rar]] was archived as [[etherpadbackup:rar]] by [[User:BryanDavis|BryanDavis]] * 05:24 [[etherpad:p/rada12dubna]] was archived as [[etherpadbackup:rada12dubna]] by [[User:BryanDavis|BryanDavis]] * 05:23 [[etherpad:p/r.d2e7710af5a20eefd93babda3f38929a]] was archived as [[etherpadbackup:r.d2e7710af5a20eefd93babda3f38929a]] by [[User:BryanDavis|BryanDavis]] * 05:22 [[etherpad:p/r.B0XlRhOdKRWT6xuH]] was archived as [[etherpadbackup:r.B0XlRhOdKRWT6xuH]] by [[User:BryanDavis|BryanDavis]] * 05:22 [[etherpad:p/quicksurveys]] was archived as [[etherpadbackup:quicksurveys]] by [[User:BryanDavis|BryanDavis]] * 05:21 [[etherpad:p/quibble-0.0.36]] was archived as [[etherpadbackup:quibble-0.0.36]] by [[User:BryanDavis|BryanDavis]] * 05:21 [[etherpad:p/quarry-wdqs-integration]] was archived as [[etherpadbackup:quarry-wdqs-integration]] by [[User:BryanDavis|BryanDavis]] * 05:20 [[etherpad:p/quarry-for-pwb-scripts]] was archived as [[etherpadbackup:quarry-for-pwb-scripts]] by [[User:BryanDavis|BryanDavis]] * 05:20 [[etherpad:p/quarry-announce-email]] was archived as [[etherpadbackup:quarry-announce-email]] by [[User:BryanDavis|BryanDavis]] * 05:19 [[etherpad:p/q_planning_retro]] was archived as [[etherpadbackup:q_planning_retro]] by [[User:BryanDavis|BryanDavis]] * 05:19 [[etherpad:p/q4planningretro]] was archived as [[etherpadbackup:q4planningretro]] by [[User:BryanDavis|BryanDavis]] * 05:18 [[etherpad:p/q4planningnotes]] was archived as [[etherpadbackup:q4planningnotes]] by [[User:BryanDavis|BryanDavis]] * 05:18 [[etherpad:p/q4_checkin]] was archived as [[etherpadbackup:q4_checkin]] by [[User:BryanDavis|BryanDavis]] * 05:17 [[etherpad:p/q2_eng_goal]] was archived as [[etherpadbackup:q2_eng_goal]] by [[User:BryanDavis|BryanDavis]] * 05:17 [[etherpad:p/python_mediawiki_devs]] was archived as [[etherpadbackup:python_mediawiki_devs]] by [[User:BryanDavis|BryanDavis]] * 05:16 [[etherpad:p/python]] was archived as [[etherpadbackup:python]] by [[User:BryanDavis|BryanDavis]] * 05:16 [[etherpad:p/push-to-deploy-beta-announce]] was archived as [[etherpadbackup:push-to-deploy-beta-announce]] by [[User:BryanDavis|BryanDavis]] * 05:15 [[etherpad:p/puppetswat]] was archived as [[etherpadbackup:puppetswat]] by [[User:BryanDavis|BryanDavis]] * 05:15 [[etherpad:p/puppetmaster-enc]] was archived as [[etherpadbackup:puppetmaster-enc]] by [[User:BryanDavis|BryanDavis]] * 05:14 [[etherpad:p/puppet_upgrade_vms]] was archived as [[etherpadbackup:puppet_upgrade_vms]] by [[User:BryanDavis|BryanDavis]] * 05:14 [[etherpad:p/puppet3]] was archived as [[etherpadbackup:puppet3]] by [[User:BryanDavis|BryanDavis]] * 05:13 [[etherpad:p/publish-new-mobile-report]] was archived as [[etherpadbackup:publish-new-mobile-report]] by [[User:BryanDavis|BryanDavis]] * 05:13 [[etherpad:p/proxy-buster]] was archived as [[etherpadbackup:proxy-buster]] by [[User:BryanDavis|BryanDavis]] * 05:12 [[etherpad:p/preciseonlabs]] was archived as [[etherpadbackup:preciseonlabs]] by [[User:BryanDavis|BryanDavis]] * 05:12 [[etherpad:p/precise-tools]] was archived as [[etherpadbackup:precise-tools]] by [[User:BryanDavis|BryanDavis]] * 05:11 [[etherpad:p/poem]] was archived as [[etherpadbackup:poem]] by [[User:BryanDavis|BryanDavis]] * 05:11 [[etherpad:p/play_store_what's_new]] was archived as [[etherpadbackup:play_store_what's_new]] by [[User:BryanDavis|BryanDavis]] * 05:10 [[etherpad:p/pii_pageviews]] was archived as [[etherpadbackup:pii_pageviews]] by [[User:BryanDavis|BryanDavis]] * 05:09 [[etherpad:p/phab_roles]] was archived as [[etherpadbackup:phab_roles]] by [[User:BryanDavis|BryanDavis]] * 05:09 [[etherpad:p/phab-2016-02-03]] was archived as [[etherpadbackup:phab-2016-02-03]] by [[User:BryanDavis|BryanDavis]] * 05:08 [[etherpad:p/phab]] was archived as [[etherpadbackup:phab]] by [[User:BryanDavis|BryanDavis]] * 05:08 [[etherpad:p/performance-team-weekly-update]] was archived as [[etherpadbackup:performance-team-weekly-update]] by [[User:BryanDavis|BryanDavis]] * 05:07 [[etherpad:p/performance-k8s]] was archived as [[etherpadbackup:performance-k8s]] by [[User:BryanDavis|BryanDavis]] * 05:07 [[etherpad:p/perfnotice]] was archived as [[etherpadbackup:perfnotice]] by [[User:BryanDavis|BryanDavis]] * 05:06 [[etherpad:p/perf-announcements]] was archived as [[etherpadbackup:perf-announcements]] by [[User:BryanDavis|BryanDavis]] * 05:06 [[etherpad:p/perf-20160223]] was archived as [[etherpadbackup:perf-20160223]] by [[User:BryanDavis|BryanDavis]] * 05:05 [[etherpad:p/paws-tools]] was archived as [[etherpadbackup:paws-tools]] by [[User:BryanDavis|BryanDavis]] * 05:05 [[etherpad:p/paws-public-url-structure]] was archived as [[etherpadbackup:paws-public-url-structure]] by [[User:BryanDavis|BryanDavis]] * 05:04 [[etherpad:p/paste]] was archived as [[etherpadbackup:paste]] by [[User:BryanDavis|BryanDavis]] * 05:04 [[etherpad:p/parsoid_docs]] was archived as [[etherpadbackup:parsoid_docs]] by [[User:BryanDavis|BryanDavis]] * 05:03 [[etherpad:p/parsoid-2019-01-16]] was archived as [[etherpadbackup:parsoid-2019-01-16]] by [[User:BryanDavis|BryanDavis]] * 05:03 [[etherpad:p/parsing_team_meeting_archive]] was archived as [[etherpadbackup:parsing_team_meeting_archive]] by [[User:BryanDavis|BryanDavis]] * 05:02 [[etherpad:p/parser-api-2018]] was archived as [[etherpadbackup:parser-api-2018]] by [[User:BryanDavis|BryanDavis]] * 05:02 [[etherpad:p/pageview_api_nodes]] was archived as [[etherpadbackup:pageview_api_nodes]] by [[User:BryanDavis|BryanDavis]] * 05:01 [[etherpad:p/pages-needing-investigation]] was archived as [[etherpadbackup:pages-needing-investigation]] by [[User:BryanDavis|BryanDavis]] * 05:01 [[etherpad:p/page-styling-zurich]] was archived as [[etherpadbackup:page-styling-zurich]] by [[User:BryanDavis|BryanDavis]] * 05:00 [[etherpad:p/otrs-migration]] was archived as [[etherpadbackup:otrs-migration]] by [[User:BryanDavis|BryanDavis]] * 05:00 [[etherpad:p/ores_task_tracking]] was archived as [[etherpadbackup:ores_task_tracking]] by [[User:BryanDavis|BryanDavis]] * 04:59 [[etherpad:p/ores]] was archived as [[etherpadbackup:ores]] by [[User:BryanDavis|BryanDavis]] * 04:58 [[etherpad:p/ops-offsite-discussions]] was archived as [[etherpadbackup:ops-offsite-discussions]] by [[User:BryanDavis|BryanDavis]] * 04:58 [[etherpad:p/ops-offsite-2016-lightning-talks]] was archived as [[etherpadbackup:ops-offsite-2016-lightning-talks]] by [[User:BryanDavis|BryanDavis]] * 04:57 [[etherpad:p/ops-offsite-2016-labstore-future]] was archived as [[etherpadbackup:ops-offsite-2016-labstore-future]] by [[User:BryanDavis|BryanDavis]] * 04:57 [[etherpad:p/opensearch-opportunities-for-improvement]] was archived as [[etherpadbackup:opensearch-opportunities-for-improvement]] by [[User:BryanDavis|BryanDavis]] * 04:56 [[etherpad:p/opendj-migration]] was archived as [[etherpadbackup:opendj-migration]] by [[User:BryanDavis|BryanDavis]] * 04:56 [[etherpad:p/open_infra_workshop]] was archived as [[etherpadbackup:open_infra_workshop]] by [[User:BryanDavis|BryanDavis]] * 04:55 [[etherpad:p/opcache-💩-2020-10-01]] was archived as [[etherpadbackup:opcache-💩-2020-10-01]] by [[User:BryanDavis|BryanDavis]] * 04:55 [[etherpad:p/oozie-restart]] was archived as [[etherpadbackup:oozie-restart]] by [[User:BryanDavis|BryanDavis]] * 04:54 [[etherpad:p/oozie]] was archived as [[etherpadbackup:oozie]] by [[User:BryanDavis|BryanDavis]] * 04:54 [[etherpad:p/omega-restarts]] was archived as [[etherpadbackup:omega-restarts]] by [[User:BryanDavis|BryanDavis]] * 04:53 [[etherpad:p/oldbastion]] was archived as [[etherpadbackup:oldbastion]] by [[User:BryanDavis|BryanDavis]] * 04:53 [[etherpad:p/okapi-dumps-2021-02-01]] was archived as [[etherpadbackup:okapi-dumps-2021-02-01]] by [[User:BryanDavis|BryanDavis]] * 04:52 [[etherpad:p/ocata-upgrade]] was archived as [[etherpadbackup:ocata-upgrade]] by [[User:BryanDavis|BryanDavis]] * 04:52 [[etherpad:p/o1Ub87jizX]] was archived as [[etherpadbackup:o1Ub87jizX]] by [[User:BryanDavis|BryanDavis]] * 04:51 [[etherpad:p/o]] was archived as [[etherpadbackup:o]] by [[User:BryanDavis|BryanDavis]] * 04:51 [[etherpad:p/nuria]] was archived as [[etherpadbackup:nuria]] by [[User:BryanDavis|BryanDavis]] * 04:50 [[etherpad:p/npp_discussion_wmhack17]] was archived as [[etherpadbackup:npp_discussion_wmhack17]] by [[User:BryanDavis|BryanDavis]] * 04:50 [[etherpad:p/nodes_with_a_public_IP]] was archived as [[etherpadbackup:nodes_with_a_public_IP]] by [[User:BryanDavis|BryanDavis]] * 04:49 [[etherpad:p/nodepool-migration]] was archived as [[etherpadbackup:nodepool-migration]] by [[User:BryanDavis|BryanDavis]] * 04:49 [[etherpad:p/night-mode-colors]] was archived as [[etherpadbackup:night-mode-colors]] by [[User:BryanDavis|BryanDavis]] * 04:48 [[etherpad:p/ng8EYRhQ7SIXy7cLBexZ]] was archived as [[etherpadbackup:ng8EYRhQ7SIXy7cLBexZ]] by [[User:BryanDavis|BryanDavis]] * 04:48 [[etherpad:p/nfsprojectchange]] was archived as [[etherpadbackup:nfsprojectchange]] by [[User:BryanDavis|BryanDavis]] * 04:47 [[etherpad:p/nfsmaint]] was archived as [[etherpadbackup:nfsmaint]] by [[User:BryanDavis|BryanDavis]] * 04:47 [[etherpad:p/nfs]] was archived as [[etherpadbackup:nfs]] by [[User:BryanDavis|BryanDavis]] * 04:46 [[etherpad:p/newton_upgrade_plan]] was archived as [[etherpadbackup:newton_upgrade_plan]] by [[User:BryanDavis|BryanDavis]] * 04:45 [[etherpad:p/newsletters-contenthandler]] was archived as [[etherpadbackup:newsletters-contenthandler]] by [[User:BryanDavis|BryanDavis]] * 04:45 [[etherpad:p/newpyter]] was archived as [[etherpadbackup:newpyter]] by [[User:BryanDavis|BryanDavis]] * 04:44 [[etherpad:p/new-webnode-toollabs]] was archived as [[etherpadbackup:new-webnode-toollabs]] by [[User:BryanDavis|BryanDavis]] * 04:44 [[etherpad:p/networkoutage]] was archived as [[etherpadbackup:networkoutage]] by [[User:BryanDavis|BryanDavis]] * 04:43 [[etherpad:p/netbox4-upgrade]] was archived as [[etherpadbackup:netbox4-upgrade]] by [[User:BryanDavis|BryanDavis]] * 04:43 [[etherpad:p/netbox-fail]] was archived as [[etherpadbackup:netbox-fail]] by [[User:BryanDavis|BryanDavis]] * 04:42 [[etherpad:p/naming-visualization]] was archived as [[etherpadbackup:naming-visualization]] by [[User:BryanDavis|BryanDavis]] * 04:42 [[etherpad:p/nTbAb5yPvBFATUylV7Vj]] was archived as [[etherpadbackup:nTbAb5yPvBFATUylV7Vj]] by [[User:BryanDavis|BryanDavis]] * 04:41 [[etherpad:p/mwtidy-checkerrors]] was archived as [[etherpadbackup:mwtidy-checkerrors]] by [[User:BryanDavis|BryanDavis]] * 04:41 [[etherpad:p/mwds15-spec-oriented-architecture]] was archived as [[etherpadbackup:mwds15-spec-oriented-architecture]] by [[User:BryanDavis|BryanDavis]] * 04:40 [[etherpad:p/mwds-profiling]] was archived as [[etherpadbackup:mwds-profiling]] by [[User:BryanDavis|BryanDavis]] * 04:40 [[etherpad:p/mw1-25-jsdeprecate]] was archived as [[etherpadbackup:mw1-25-jsdeprecate]] by [[User:BryanDavis|BryanDavis]] * 04:39 [[etherpad:p/mw-php70-tests]] was archived as [[etherpadbackup:mw-php70-tests]] by [[User:BryanDavis|BryanDavis]] * 04:39 [[etherpad:p/mvrelease]] was archived as [[etherpadbackup:mvrelease]] by [[User:BryanDavis|BryanDavis]] * 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-12-24]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-12-24]] by [[User:BryanDavis|BryanDavis]] * 04:38 [[etherpad:p/multimedia-weekly-meeting-2015-01-14]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2015-01-14]] by [[User:BryanDavis|BryanDavis]] * 04:37 [[etherpad:p/multimedia-weekly-meeting-2014-01-06]] was archived as [[etherpadbackup:multimedia-weekly-meeting-2014-01-06]] by [[User:BryanDavis|BryanDavis]] * 04:37 [[etherpad:p/multimedia-mmv-technical-retrospective]] was archived as [[etherpadbackup:multimedia-mmv-technical-retrospective]] by [[User:BryanDavis|BryanDavis]] * 04:36 [[etherpad:p/multimedia-design-05-15-2014]] was archived as [[etherpadbackup:multimedia-design-05-15-2014]] by [[User:BryanDavis|BryanDavis]] * 04:36 [[etherpad:p/multimedia-analytics-queries]] was archived as [[etherpadbackup:multimedia-analytics-queries]] by [[User:BryanDavis|BryanDavis]] * 04:35 [[etherpad:p/morethan10g]] was archived as [[etherpadbackup:morethan10g]] by [[User:BryanDavis|BryanDavis]] * 04:35 [[etherpad:p/monitoring-lvs-l2]] was archived as [[etherpadbackup:monitoring-lvs-l2]] by [[User:BryanDavis|BryanDavis]] * 04:34 [[etherpad:p/mod-conf]] was archived as [[etherpadbackup:mod-conf]] by [[User:BryanDavis|BryanDavis]] * 04:34 [[etherpad:p/mobileweb-wsj]] was archived as [[etherpadbackup:mobileweb-wsj]] by [[User:BryanDavis|BryanDavis]] * 04:33 [[etherpad:p/mobilesprintname]] was archived as [[etherpadbackup:mobilesprintname]] by [[User:BryanDavis|BryanDavis]] * 04:33 [[etherpad:p/mobilenav-sprint2-retro]] was archived as [[etherpadbackup:mobilenav-sprint2-retro]] by [[User:BryanDavis|BryanDavis]] * 04:32 [[etherpad:p/mobilenav-sprint1-retro]] was archived as [[etherpadbackup:mobilenav-sprint1-retro]] by [[User:BryanDavis|BryanDavis]] * 04:32 [[etherpad:p/mobileappplanningQ32014]] was archived as [[etherpadbackup:mobileappplanningQ32014]] by [[User:BryanDavis|BryanDavis]] * 04:31 [[etherpad:p/mobile_web_q3]] was archived as [[etherpadbackup:mobile_web_q3]] by [[User:BryanDavis|BryanDavis]] * 04:30 [[etherpad:p/mobile_web_health_check]] was archived as [[etherpadbackup:mobile_web_health_check]] by [[User:BryanDavis|BryanDavis]] * 04:30 [[etherpad:p/mobile_web_design_review]] was archived as [[etherpadbackup:mobile_web_design_review]] by [[User:BryanDavis|BryanDavis]] * 04:29 [[etherpad:p/mobile_web_breakout]] was archived as [[etherpadbackup:mobile_web_breakout]] by [[User:BryanDavis|BryanDavis]] * 04:29 [[etherpad:p/mobile_apps_onboarding_brian]] was archived as [[etherpadbackup:mobile_apps_onboarding_brian]] by [[User:BryanDavis|BryanDavis]] * 04:28 [[etherpad:p/mobile-users-faq]] was archived as [[etherpadbackup:mobile-users-faq]] by [[User:BryanDavis|BryanDavis]] * 04:28 [[etherpad:p/mobile-tutorial-practice]] was archived as [[etherpadbackup:mobile-tutorial-practice]] by [[User:BryanDavis|BryanDavis]] * 04:27 [[etherpad:p/mobile-retrospective-q3-2013]] was archived as [[etherpadbackup:mobile-retrospective-q3-2013]] by [[User:BryanDavis|BryanDavis]] * 04:27 [[etherpad:p/mobile-retrospective-20130128]] was archived as [[etherpadbackup:mobile-retrospective-20130128]] by [[User:BryanDavis|BryanDavis]] * 04:26 [[etherpad:p/mobile-retrospective-20130114]] was archived as [[etherpadbackup:mobile-retrospective-20130114]] by [[User:BryanDavis|BryanDavis]] * 04:26 [[etherpad:p/mobile-retrospective-20121217]] was archived as [[etherpadbackup:mobile-retrospective-20121217]] by [[User:BryanDavis|BryanDavis]] * 04:25 [[etherpad:p/mobile-retrospective-20120102]] was archived as [[etherpadbackup:mobile-retrospective-20120102]] by [[User:BryanDavis|BryanDavis]] * 04:25 [[etherpad:p/mobile-q4-planning-2013]] was archived as [[etherpadbackup:mobile-q4-planning-2013]] by [[User:BryanDavis|BryanDavis]] * 04:24 [[etherpad:p/mobile-q3-planning-2013]] was archived as [[etherpadbackup:mobile-q3-planning-2013]] by [[User:BryanDavis|BryanDavis]] * 04:24 [[etherpad:p/mobile-ops-syncup-28feb2013]] was archived as [[etherpadbackup:mobile-ops-syncup-28feb2013]] by [[User:BryanDavis|BryanDavis]] * 04:23 [[etherpad:p/mobile-ime-support]] was archived as [[etherpadbackup:mobile-ime-support]] by [[User:BryanDavis|BryanDavis]] * 04:23 [[etherpad:p/mobile-campaigns]] was archived as [[etherpadbackup:mobile-campaigns]] by [[User:BryanDavis|BryanDavis]] * 04:22 [[etherpad:p/mobile-app-workflow-el-1]] was archived as [[etherpadbackup:mobile-app-workflow-el-1]] by [[User:BryanDavis|BryanDavis]] * 04:22 [[etherpad:p/mobile-app-versions]] was archived as [[etherpadbackup:mobile-app-versions]] by [[User:BryanDavis|BryanDavis]] * 04:21 [[etherpad:p/mobile-app-otrs-feedback]] was archived as [[etherpadbackup:mobile-app-otrs-feedback]] by [[User:BryanDavis|BryanDavis]] * 04:21 [[etherpad:p/mobile-app-dashboard]] was archived as [[etherpadbackup:mobile-app-dashboard]] by [[User:BryanDavis|BryanDavis]] * 04:20 [[etherpad:p/mmv-text-size-changes]] was archived as [[etherpadbackup:mmv-text-size-changes]] by [[User:BryanDavis|BryanDavis]] * 04:20 [[etherpad:p/mmodell]] was archived as [[etherpadbackup:mmodell]] by [[User:BryanDavis|BryanDavis]] * 04:19 [[etherpad:p/mm-standup-notes]] was archived as [[etherpadbackup:mm-standup-notes]] by [[User:BryanDavis|BryanDavis]] * 04:19 [[etherpad:p/misc-migration]] was archived as [[etherpadbackup:misc-migration]] by [[User:BryanDavis|BryanDavis]] * 04:18 [[etherpad:p/migrate-k8s-etcd]] was archived as [[etherpadbackup:migrate-k8s-etcd]] by [[User:BryanDavis|BryanDavis]] * 04:18 [[etherpad:p/mf]] was archived as [[etherpadbackup:mf]] by [[User:BryanDavis|BryanDavis]] * 04:17 [[etherpad:p/mediawiki-vagrant]] was archived as [[etherpadbackup:mediawiki-vagrant]] by [[User:BryanDavis|BryanDavis]] * 04:16 [[etherpad:p/mazza]] was archived as [[etherpadbackup:mazza]] by [[User:BryanDavis|BryanDavis]] * 04:16 [[etherpad:p/mayankmadan]] was archived as [[etherpadbackup:mayankmadan]] by [[User:BryanDavis|BryanDavis]] * 04:15 [[etherpad:p/mailman-Aug-2015]] was archived as [[etherpadbackup:mailman-Aug-2015]] by [[User:BryanDavis|BryanDavis]] * 04:15 [[etherpad:p/magru_server_swaps]] was archived as [[etherpadbackup:magru_server_swaps]] by [[User:BryanDavis|BryanDavis]] * 04:14 [[etherpad:p/magnumusersfeb2026]] was archived as [[etherpadbackup:magnumusersfeb2026]] by [[User:BryanDavis|BryanDavis]] * 04:14 [[etherpad:p/mBBgYyJeBG]] was archived as [[etherpadbackup:mBBgYyJeBG]] by [[User:BryanDavis|BryanDavis]] * 04:13 [[etherpad:p/m1-service-owners]] was archived as [[etherpadbackup:m1-service-owners]] by [[User:BryanDavis|BryanDavis]] * 04:13 [[etherpad:p/lyon-service-template]] was archived as [[etherpadbackup:lyon-service-template]] by [[User:BryanDavis|BryanDavis]] * 04:12 [[etherpad:p/lucid_mini_hack]] was archived as [[etherpadbackup:lucid_mini_hack]] by [[User:BryanDavis|BryanDavis]] * 04:12 [[etherpad:p/lubaochuan]] was archived as [[etherpadbackup:lubaochuan]] by [[User:BryanDavis|BryanDavis]] * 04:11 [[etherpad:p/logchange]] was archived as [[etherpadbackup:logchange]] by [[User:BryanDavis|BryanDavis]] * 04:11 [[etherpad:p/linkpass]] was archived as [[etherpadbackup:linkpass]] by [[User:BryanDavis|BryanDavis]] * 04:10 [[etherpad:p/legal-text-upload-dialog]] was archived as [[etherpadbackup:legal-text-upload-dialog]] by [[User:BryanDavis|BryanDavis]] * 04:10 [[etherpad:p/language-and-localization]] was archived as [[etherpadbackup:language-and-localization]] by [[User:BryanDavis|BryanDavis]] * 04:09 [[etherpad:p/langaugescreenshot-upload-error]] was archived as [[etherpadbackup:langaugescreenshot-upload-error]] by [[User:BryanDavis|BryanDavis]] * 04:09 [[etherpad:p/lang-bar]] was archived as [[etherpadbackup:lang-bar]] by [[User:BryanDavis|BryanDavis]] * 04:08 [[etherpad:p/labwebneedsports]] was archived as [[etherpadbackup:labwebneedsports]] by [[User:BryanDavis|BryanDavis]] * 04:08 [[etherpad:p/labweb]] was archived as [[etherpadbackup:labweb]] by [[User:BryanDavis|BryanDavis]] * 04:07 [[etherpad:p/labvirt_reboots]] was archived as [[etherpadbackup:labvirt_reboots]] by [[User:BryanDavis|BryanDavis]] * 04:07 [[etherpad:p/labvirt1015]] was archived as [[etherpadbackup:labvirt1015]] by [[User:BryanDavis|BryanDavis]] * 04:06 [[etherpad:p/labvirt1003instances]] was archived as [[etherpadbackup:labvirt1003instances]] by [[User:BryanDavis|BryanDavis]] * 04:06 [[etherpad:p/labtes_mitaka_upgrade_notes]] was archived as [[etherpadbackup:labtes_mitaka_upgrade_notes]] by [[User:BryanDavis|BryanDavis]] * 04:05 [[etherpad:p/labstore10045-move]] was archived as [[etherpadbackup:labstore10045-move]] by [[User:BryanDavis|BryanDavis]] * 04:04 [[etherpad:p/labservices_failover]] was archived as [[etherpadbackup:labservices_failover]] by [[User:BryanDavis|BryanDavis]] * 04:04 [[etherpad:p/labsdb10067]] was archived as [[etherpadbackup:labsdb10067]] by [[User:BryanDavis|BryanDavis]] * 04:03 [[etherpad:p/labsdb-replication]] was archived as [[etherpadbackup:labsdb-replication]] by [[User:BryanDavis|BryanDavis]] * 04:03 [[etherpad:p/labsdb-goal]] was archived as [[etherpadbackup:labsdb-goal]] by [[User:BryanDavis|BryanDavis]] * 04:02 [[etherpad:p/labsdb-chat-july-18]] was archived as [[etherpadbackup:labsdb-chat-july-18]] by [[User:BryanDavis|BryanDavis]] * 04:02 [[etherpad:p/labs_migration]] was archived as [[etherpadbackup:labs_migration]] by [[User:BryanDavis|BryanDavis]] * 04:01 [[etherpad:p/labs-survey-interview-questions]] was archived as [[etherpadbackup:labs-survey-interview-questions]] by [[User:BryanDavis|BryanDavis]] * 04:01 [[etherpad:p/labs-report-q4-1]] was archived as [[etherpadbackup:labs-report-q4-1]] by [[User:BryanDavis|BryanDavis]] * 04:00 [[etherpad:p/labs-releng-k8s-confluence]] was archived as [[etherpadbackup:labs-releng-k8s-confluence]] by [[User:BryanDavis|BryanDavis]] * 04:00 [[etherpad:p/labs-nfs-20140413-nfsoutage]] was archived as [[etherpadbackup:labs-nfs-20140413-nfsoutage]] by [[User:BryanDavis|BryanDavis]] * 03:59 [[etherpad:p/labs-migration]] was archived as [[etherpadbackup:labs-migration]] by [[User:BryanDavis|BryanDavis]] * 03:59 [[etherpad:p/labs-meeting-apr-25-2016]] was archived as [[etherpadbackup:labs-meeting-apr-25-2016]] by [[User:BryanDavis|BryanDavis]] * 03:58 [[etherpad:p/labs-incident-timeline]] was archived as [[etherpadbackup:labs-incident-timeline]] by [[User:BryanDavis|BryanDavis]] * 03:58 [[etherpad:p/labs-cleanup-2015]] was archived as [[etherpadbackup:labs-cleanup-2015]] by [[User:BryanDavis|BryanDavis]] * 03:57 [[etherpad:p/labpuppetmaster1001-stragglers]] was archived as [[etherpadbackup:labpuppetmaster1001-stragglers]] by [[User:BryanDavis|BryanDavis]] * 03:57 [[etherpad:p/labnetfailover]] was archived as [[etherpadbackup:labnetfailover]] by [[User:BryanDavis|BryanDavis]] * 03:56 [[etherpad:p/l10nupdate]] was archived as [[etherpadbackup:l10nupdate]] by [[User:BryanDavis|BryanDavis]] * 03:56 [[etherpad:p/l10n-translate-centralnotice]] was archived as [[etherpadbackup:l10n-translate-centralnotice]] by [[User:BryanDavis|BryanDavis]] * 03:55 [[etherpad:p/l10n-team-2013-10]] was archived as [[etherpadbackup:l10n-team-2013-10]] by [[User:BryanDavis|BryanDavis]] * 03:55 [[etherpad:p/kubecon-2017-offsite-agenda]] was archived as [[etherpadbackup:kubecon-2017-offsite-agenda]] by [[User:BryanDavis|BryanDavis]] * 03:54 [[etherpad:p/kubecon]] was archived as [[etherpadbackup:kubecon]] by [[User:BryanDavis|BryanDavis]] * 03:54 [[etherpad:p/kubeadm]] was archived as [[etherpadbackup:kubeadm]] by [[User:BryanDavis|BryanDavis]] * 03:53 [[etherpad:p/kill-gridengine-slowly]] was archived as [[etherpadbackup:kill-gridengine-slowly]] by [[User:BryanDavis|BryanDavis]] * 03:53 [[etherpad:p/keystoneldap]] was archived as [[etherpadbackup:keystoneldap]] by [[User:BryanDavis|BryanDavis]] * 03:52 [[etherpad:p/keystonebootstrap]] was archived as [[etherpadbackup:keystonebootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:52 [[etherpad:p/kafkaswift]] was archived as [[etherpadbackup:kafkaswift]] by [[User:BryanDavis|BryanDavis]] * 03:51 [[etherpad:p/kZaM0swM4c]] was archived as [[etherpadbackup:kZaM0swM4c]] by [[User:BryanDavis|BryanDavis]] * 03:51 [[etherpad:p/kAh7UZSoyI]] was archived as [[etherpadbackup:kAh7UZSoyI]] by [[User:BryanDavis|BryanDavis]] * 03:50 [[etherpad:p/kAKhhwCsJ7]] was archived as [[etherpadbackup:kAKhhwCsJ7]] by [[User:BryanDavis|BryanDavis]] * 03:49 [[etherpad:p/k8s-status-wishlist]] was archived as [[etherpadbackup:k8s-status-wishlist]] by [[User:BryanDavis|BryanDavis]] * 03:49 [[etherpad:p/jupyter-at-wikimedia]] was archived as [[etherpadbackup:jupyter-at-wikimedia]] by [[User:BryanDavis|BryanDavis]] * 03:48 [[etherpad:p/junk]] was archived as [[etherpadbackup:junk]] by [[User:BryanDavis|BryanDavis]] * 03:48 [[etherpad:p/joes-random-notes]] was archived as [[etherpadbackup:joes-random-notes]] by [[User:BryanDavis|BryanDavis]] * 03:47 [[etherpad:p/jmxtrans-packaging-improvements]] was archived as [[etherpadbackup:jmxtrans-packaging-improvements]] by [[User:BryanDavis|BryanDavis]] * 03:47 [[etherpad:p/jjb]] was archived as [[etherpadbackup:jjb]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/jenkins-upgrading]] was archived as [[etherpadbackup:jenkins-upgrading]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/jenkins-job-builder]] was archived as [[etherpadbackup:jenkins-job-builder]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/javascript-scoping]] was archived as [[etherpadbackup:javascript-scoping]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/jagori]] was archived as [[etherpadbackup:jagori]] by [[User:BryanDavis|BryanDavis]] * 03:44 [[etherpad:p/jQUprade]] was archived as [[etherpadbackup:jQUprade]] by [[User:BryanDavis|BryanDavis]] * 03:44 [[etherpad:p/j9qaVR_wF9bIWn6BWuIr]] was archived as [[etherpadbackup:j9qaVR_wF9bIWn6BWuIr]] by [[User:BryanDavis|BryanDavis]] * 03:43 [[etherpad:p/j4Obl63I8y]] was archived as [[etherpadbackup:j4Obl63I8y]] by [[User:BryanDavis|BryanDavis]] * 03:43 [[etherpad:p/iterationname]] was archived as [[etherpadbackup:iterationname]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/is]] was archived as [[etherpadbackup:is]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/ipv6]] was archived as [[etherpadbackup:ipv6]] by [[User:BryanDavis|BryanDavis]] * 03:41 [[etherpad:p/iosreleasing]] was archived as [[etherpadbackup:iosreleasing]] by [[User:BryanDavis|BryanDavis]] * 03:41 [[etherpad:p/iosrelease]] was archived as [[etherpadbackup:iosrelease]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/ios_app_update_release_notes]] was archived as [[etherpadbackup:ios_app_update_release_notes]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/ios-5.0]] was archived as [[etherpadbackup:ios-5.0]] by [[User:BryanDavis|BryanDavis]] * 03:39 [[etherpad:p/ios-3-1beta1-release-notes]] was archived as [[etherpadbackup:ios-3-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 03:39 [[etherpad:p/intro_to_revscores]] was archived as [[etherpadbackup:intro_to_revscores]] by [[User:BryanDavis|BryanDavis]] * 03:38 [[etherpad:p/internal-graph-split-lvs]] was archived as [[etherpadbackup:internal-graph-split-lvs]] by [[User:BryanDavis|BryanDavis]] * 03:38 [[etherpad:p/intercontinental-debugging-may-07-2012]] was archived as [[etherpadbackup:intercontinental-debugging-may-07-2012]] by [[User:BryanDavis|BryanDavis]] * 03:37 [[etherpad:p/incident-20190509-codfwpuppetmasterdown]] was archived as [[etherpadbackup:incident-20190509-codfwpuppetmasterdown]] by [[User:BryanDavis|BryanDavis]] * 03:37 [[etherpad:p/in_article_search_query]] was archived as [[etherpadbackup:in_article_search_query]] by [[User:BryanDavis|BryanDavis]] * 03:36 [[etherpad:p/import_wcqs]] was archived as [[etherpadbackup:import_wcqs]] by [[User:BryanDavis|BryanDavis]] * 03:36 [[etherpad:p/ihatesasl]] was archived as [[etherpadbackup:ihatesasl]] by [[User:BryanDavis|BryanDavis]] * 03:35 [[etherpad:p/iOS_available_locale_ids]] was archived as [[etherpadbackup:iOS_available_locale_ids]] by [[User:BryanDavis|BryanDavis]] * 03:34 [[etherpad:p/iOSDataMerge]] was archived as [[etherpadbackup:iOSDataMerge]] by [[User:BryanDavis|BryanDavis]] * 03:34 [[etherpad:p/iOSAppi18nIssues]] was archived as [[etherpadbackup:iOSAppi18nIssues]] by [[User:BryanDavis|BryanDavis]] * 03:33 [[etherpad:p/iOSAppDescription]] was archived as [[etherpadbackup:iOSAppDescription]] by [[User:BryanDavis|BryanDavis]] * 03:33 [[etherpad:p/iOS-ChangeLog]] was archived as [[etherpadbackup:iOS-ChangeLog]] by [[User:BryanDavis|BryanDavis]] * 03:32 [[etherpad:p/iOS-7-issues]] was archived as [[etherpadbackup:iOS-7-issues]] by [[User:BryanDavis|BryanDavis]] * 03:32 [[etherpad:p/i8XjsnBxxh]] was archived as [[etherpadbackup:i8XjsnBxxh]] by [[User:BryanDavis|BryanDavis]] * 03:31 [[etherpad:p/i18n-rfc-2013-11]] was archived as [[etherpadbackup:i18n-rfc-2013-11]] by [[User:BryanDavis|BryanDavis]] * 03:31 [[etherpad:p/huggle_section]] was archived as [[etherpadbackup:huggle_section]] by [[User:BryanDavis|BryanDavis]] * 03:30 [[etherpad:p/horizonsso]] was archived as [[etherpadbackup:horizonsso]] by [[User:BryanDavis|BryanDavis]] * 03:30 [[etherpad:p/hi6ZshFUi2]] was archived as [[etherpadbackup:hi6ZshFUi2]] by [[User:BryanDavis|BryanDavis]] * 03:29 [[etherpad:p/hhvm]] was archived as [[etherpadbackup:hhvm]] by [[User:BryanDavis|BryanDavis]] * 03:29 [[etherpad:p/helm-toollabs]] was archived as [[etherpadbackup:helm-toollabs]] by [[User:BryanDavis|BryanDavis]] * 03:28 [[etherpad:p/helm-chart-development]] was archived as [[etherpadbackup:helm-chart-development]] by [[User:BryanDavis|BryanDavis]] * 03:28 [[etherpad:p/handoff-wdqs-T388134]] was archived as [[etherpadbackup:handoff-wdqs-T388134]] by [[User:BryanDavis|BryanDavis]] * 03:27 [[etherpad:p/hackathon-showcase-before-the-catastrophe]] was archived as [[etherpadbackup:hackathon-showcase-before-the-catastrophe]] by [[User:BryanDavis|BryanDavis]] * 03:27 [[etherpad:p/hackathon-rhinosf1-barriers]] was archived as [[etherpadbackup:hackathon-rhinosf1-barriers]] by [[User:BryanDavis|BryanDavis]] * 03:26 [[etherpad:p/gridengine-uses]] was archived as [[etherpadbackup:gridengine-uses]] by [[User:BryanDavis|BryanDavis]] * 03:26 [[etherpad:p/grid-backup]] was archived as [[etherpadbackup:grid-backup]] by [[User:BryanDavis|BryanDavis]] * 03:25 [[etherpad:p/graphitetodo]] was archived as [[etherpadbackup:graphitetodo]] by [[User:BryanDavis|BryanDavis]] * 03:25 [[etherpad:p/grantreview]] was archived as [[etherpadbackup:grantreview]] by [[User:BryanDavis|BryanDavis]] * 03:24 [[etherpad:p/gitlab-sync]] was archived as [[etherpadbackup:gitlab-sync]] by [[User:BryanDavis|BryanDavis]] * 03:24 [[etherpad:p/git]] was archived as [[etherpadbackup:git]] by [[User:BryanDavis|BryanDavis]] * 03:23 [[etherpad:p/gerrit-4841]] was archived as [[etherpadbackup:gerrit-4841]] by [[User:BryanDavis|BryanDavis]] * 03:23 [[etherpad:p/gdash-labels-clarification]] was archived as [[etherpadbackup:gdash-labels-clarification]] by [[User:BryanDavis|BryanDavis]] * 03:22 [[etherpad:p/gcimail]] was archived as [[etherpadbackup:gcimail]] by [[User:BryanDavis|BryanDavis]] * 03:22 [[etherpad:p/gannum]] was archived as [[etherpadbackup:gannum]] by [[User:BryanDavis|BryanDavis]] * 03:21 [[etherpad:p/g5an_WR4fa02lVjmHQse]] was archived as [[etherpadbackup:g5an_WR4fa02lVjmHQse]] by [[User:BryanDavis|BryanDavis]] * 03:21 [[etherpad:p/g505410announce]] was archived as [[etherpadbackup:g505410announce]] by [[User:BryanDavis|BryanDavis]] * 03:20 [[etherpad:p/g4-flavors]] was archived as [[etherpadbackup:g4-flavors]] by [[User:BryanDavis|BryanDavis]] * 03:19 [[etherpad:p/future-of-language-converter]] was archived as [[etherpadbackup:future-of-language-converter]] by [[User:BryanDavis|BryanDavis]] * 03:19 [[etherpad:p/fullstack_leaks]] was archived as [[etherpadbackup:fullstack_leaks]] by [[User:BryanDavis|BryanDavis]] * 03:18 [[etherpad:p/fullstack-flapping]] was archived as [[etherpadbackup:fullstack-flapping]] by [[User:BryanDavis|BryanDavis]] * 03:18 [[etherpad:p/fr-tech_chores_2023_fish_HEAD^]] was archived as [[etherpadbackup:fr-tech_chores_2023_fish_HEAD^]] by [[User:BryanDavis|BryanDavis]] * 03:17 [[etherpad:p/fp-Hrk0fve_GXMeMZFJk]] was archived as [[etherpadbackup:fp-Hrk0fve_GXMeMZFJk]] by [[User:BryanDavis|BryanDavis]] * 03:17 [[etherpad:p/floating-ip-aliaser]] was archived as [[etherpadbackup:floating-ip-aliaser]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/flagellating-funnel]] was archived as [[etherpadbackup:flagellating-funnel]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/extdist]] was archived as [[etherpadbackup:extdist]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/existensions]] was archived as [[etherpadbackup:existensions]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/eventlogging_stag]] was archived as [[etherpadbackup:eventlogging_stag]] by [[User:BryanDavis|BryanDavis]] * 03:14 [[etherpad:p/eventbus-statsd]] was archived as [[etherpadbackup:eventbus-statsd]] by [[User:BryanDavis|BryanDavis]] * 03:14 [[etherpad:p/essex-upgrade]] was archived as [[etherpadbackup:essex-upgrade]] by [[User:BryanDavis|BryanDavis]] * 03:13 [[etherpad:p/esams-followup]] was archived as [[etherpadbackup:esams-followup]] by [[User:BryanDavis|BryanDavis]] * 03:13 [[etherpad:p/eqiad1-upgrade-pike]] was archived as [[etherpadbackup:eqiad1-upgrade-pike]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/eqiad1-neutron-bootstrap]] was archived as [[etherpadbackup:eqiad1-neutron-bootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/eqiad1-keystone-bootstrap]] was archived as [[etherpadbackup:eqiad1-keystone-bootstrap]] by [[User:BryanDavis|BryanDavis]] * 03:11 [[etherpad:p/eqiad1]] was archived as [[etherpadbackup:eqiad1]] by [[User:BryanDavis|BryanDavis]] * 03:11 [[etherpad:p/enwikiSOPADBUpgrades]] was archived as [[etherpadbackup:enwikiSOPADBUpgrades]] by [[User:BryanDavis|BryanDavis]] * 03:10 [[etherpad:p/engprod-irc]] was archived as [[etherpadbackup:engprod-irc]] by [[User:BryanDavis|BryanDavis]] * 03:10 [[etherpad:p/empty_vps_projects]] was archived as [[etherpadbackup:empty_vps_projects]] by [[User:BryanDavis|BryanDavis]] * 03:09 [[etherpad:p/embeddings_and_topic_models]] was archived as [[etherpadbackup:embeddings_and_topic_models]] by [[User:BryanDavis|BryanDavis]] * 03:09 [[etherpad:p/email-tools]] was archived as [[etherpadbackup:email-tools]] by [[User:BryanDavis|BryanDavis]] * 03:08 [[etherpad:p/elukey-netflow]] was archived as [[etherpadbackup:elukey-netflow]] by [[User:BryanDavis|BryanDavis]] * 03:07 [[etherpad:p/elastic-single-node-testing]] was archived as [[etherpadbackup:elastic-single-node-testing]] by [[User:BryanDavis|BryanDavis]] * 03:07 [[etherpad:p/elastic-2-opensearch-T388610]] was archived as [[etherpadbackup:elastic-2-opensearch-T388610]] by [[User:BryanDavis|BryanDavis]] * 03:06 [[etherpad:p/el_utils]] was archived as [[etherpadbackup:el_utils]] by [[User:BryanDavis|BryanDavis]] * 03:06 [[etherpad:p/eiwGl_RxtBIFSyuxv5Yr]] was archived as [[etherpadbackup:eiwGl_RxtBIFSyuxv5Yr]] by [[User:BryanDavis|BryanDavis]] * 03:05 [[etherpad:p/effiee]] was archived as [[etherpadbackup:effiee]] by [[User:BryanDavis|BryanDavis]] * 03:05 [[etherpad:p/editing_logging]] was archived as [[etherpadbackup:editing_logging]] by [[User:BryanDavis|BryanDavis]] * 03:04 [[etherpad:p/edit_types_event_schema]] was archived as [[etherpadbackup:edit_types_event_schema]] by [[User:BryanDavis|BryanDavis]] * 03:04 [[etherpad:p/edit_history_vetting]] was archived as [[etherpadbackup:edit_history_vetting]] by [[User:BryanDavis|BryanDavis]] * 03:03 [[etherpad:p/e1]] was archived as [[etherpadbackup:e1]] by [[User:BryanDavis|BryanDavis]] * 03:03 [[etherpad:p/e]] was archived as [[etherpadbackup:e]] by [[User:BryanDavis|BryanDavis]] * 03:02 [[etherpad:p/doc-from-git]] was archived as [[etherpadbackup:doc-from-git]] by [[User:BryanDavis|BryanDavis]] * 03:02 [[etherpad:p/dnsthings]] was archived as [[etherpadbackup:dnsthings]] by [[User:BryanDavis|BryanDavis]] * 03:01 [[etherpad:p/dns_cleanup]] was archived as [[etherpadbackup:dns_cleanup]] by [[User:BryanDavis|BryanDavis]] * 03:01 [[etherpad:p/dmitry-onboarding]] was archived as [[etherpadbackup:dmitry-onboarding]] by [[User:BryanDavis|BryanDavis]] * 03:00 [[etherpad:p/disk_aio_setting]] was archived as [[etherpadbackup:disk_aio_setting]] by [[User:BryanDavis|BryanDavis]] * 03:00 [[etherpad:p/diff_test]] was archived as [[etherpadbackup:diff_test]] by [[User:BryanDavis|BryanDavis]] * 02:59 [[etherpad:p/diamond-deployment]] was archived as [[etherpadbackup:diamond-deployment]] by [[User:BryanDavis|BryanDavis]] * 02:59 [[etherpad:p/dhcp-option82-sunset]] was archived as [[etherpadbackup:dhcp-option82-sunset]] by [[User:BryanDavis|BryanDavis]] * 02:58 [[etherpad:p/devsummit18-thirdpartymediawiki]] was archived as [[etherpadbackup:devsummit18-thirdpartymediawiki]] by [[User:BryanDavis|BryanDavis]] * 02:58 [[etherpad:p/devsummit17-support-global-preferences]] was archived as [[etherpadbackup:devsummit17-support-global-preferences]] by [[User:BryanDavis|BryanDavis]] * 02:57 [[etherpad:p/devsummit17-scaling-database-schema]] was archived as [[etherpadbackup:devsummit17-scaling-database-schema]] by [[User:BryanDavis|BryanDavis]] * 02:57 [[etherpad:p/devsummit17-integrating-mediawiki]] was archived as [[etherpadbackup:devsummit17-integrating-mediawiki]] by [[User:BryanDavis|BryanDavis]] * 02:56 [[etherpad:p/devsummit17-algorithmic-dangers]] was archived as [[etherpadbackup:devsummit17-algorithmic-dangers]] by [[User:BryanDavis|BryanDavis]] * 02:56 [[etherpad:p/devsummit17-ai-wishlist]] was archived as [[etherpadbackup:devsummit17-ai-wishlist]] by [[User:BryanDavis|BryanDavis]] * 02:55 [[etherpad:p/devsummit17-T149624]] was archived as [[etherpadbackup:devsummit17-T149624]] by [[User:BryanDavis|BryanDavis]] * 02:55 [[etherpad:p/devsummit17-RedesignSpecialSearchPage]] was archived as [[etherpadbackup:devsummit17-RedesignSpecialSearchPage]] by [[User:BryanDavis|BryanDavis]] * 02:54 [[etherpad:p/devsummit17-ProjectOREO]] was archived as [[etherpadbackup:devsummit17-ProjectOREO]] by [[User:BryanDavis|BryanDavis]] * 02:53 [[etherpad:p/devsummit17-CodeReview]] was archived as [[etherpadbackup:devsummit17-CodeReview]] by [[User:BryanDavis|BryanDavis]] * 02:53 [[etherpad:p/devpowpow-2015-02-25]] was archived as [[etherpadbackup:devpowpow-2015-02-25]] by [[User:BryanDavis|BryanDavis]] * 02:52 [[etherpad:p/design_review_10_22]] was archived as [[etherpadbackup:design_review_10_22]] by [[User:BryanDavis|BryanDavis]] * 02:52 [[etherpad:p/design-thingies-jan-android]] was archived as [[etherpadbackup:design-thingies-jan-android]] by [[User:BryanDavis|BryanDavis]] * 02:51 [[etherpad:p/design]] was archived as [[etherpadbackup:design]] by [[User:BryanDavis|BryanDavis]] * 02:51 [[etherpad:p/deprecate-precise]] was archived as [[etherpadbackup:deprecate-precise]] by [[User:BryanDavis|BryanDavis]] * 02:50 [[etherpad:p/depool]] was archived as [[etherpadbackup:depool]] by [[User:BryanDavis|BryanDavis]] * 02:50 [[etherpad:p/deployworkinggroup]] was archived as [[etherpadbackup:deployworkinggroup]] by [[User:BryanDavis|BryanDavis]] * 02:49 [[etherpad:p/deploytrainingprep]] was archived as [[etherpadbackup:deploytrainingprep]] by [[User:BryanDavis|BryanDavis]] * 02:49 [[etherpad:p/deploy-20150427-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150427-SWAT-evening]] by [[User:BryanDavis|BryanDavis]] * 02:48 [[etherpad:p/deploy-20150416-SWAT-evening]] was archived as [[etherpadbackup:deploy-20150416-SWAT-evening]] by [[User:BryanDavis|BryanDavis]] * 02:48 [[etherpad:p/deletedesedomains]] was archived as [[etherpadbackup:deletedesedomains]] by [[User:BryanDavis|BryanDavis]] * 02:47 [[etherpad:p/dear-tool-user]] was archived as [[etherpadbackup:dear-tool-user]] by [[User:BryanDavis|BryanDavis]] * 02:47 [[etherpad:p/dchanPlate2015]] was archived as [[etherpadbackup:dchanPlate2015]] by [[User:BryanDavis|BryanDavis]] * 02:46 [[etherpad:p/db-renames]] was archived as [[etherpadbackup:db-renames]] by [[User:BryanDavis|BryanDavis]] * 02:46 [[etherpad:p/datalossjuly2]] was archived as [[etherpadbackup:datalossjuly2]] by [[User:BryanDavis|BryanDavis]] * 02:45 [[etherpad:p/cx-presentation]] was archived as [[etherpadbackup:cx-presentation]] by [[User:BryanDavis|BryanDavis]] * 02:45 [[etherpad:p/cx-markup-alignment-uppercase]] was archived as [[etherpadbackup:cx-markup-alignment-uppercase]] by [[User:BryanDavis|BryanDavis]] * 02:44 [[etherpad:p/cx-markup-alignment]] was archived as [[etherpadbackup:cx-markup-alignment]] by [[User:BryanDavis|BryanDavis]] * 02:44 [[etherpad:p/cx-invite]] was archived as [[etherpadbackup:cx-invite]] by [[User:BryanDavis|BryanDavis]] * 02:43 [[etherpad:p/cx-hack]] was archived as [[etherpadbackup:cx-hack]] by [[User:BryanDavis|BryanDavis]] * 02:43 [[etherpad:p/cx-composer]] was archived as [[etherpadbackup:cx-composer]] by [[User:BryanDavis|BryanDavis]] * 02:42 [[etherpad:p/csKD3No9oL]] was archived as [[etherpadbackup:csKD3No9oL]] by [[User:BryanDavis|BryanDavis]] * 02:42 [[etherpad:p/cronerror]] was archived as [[etherpadbackup:cronerror]] by [[User:BryanDavis|BryanDavis]] * 02:41 [[etherpad:p/cr2-codfw_fpc0]] was archived as [[etherpadbackup:cr2-codfw_fpc0]] by [[User:BryanDavis|BryanDavis]] * 02:41 [[etherpad:p/conftool-T379329]] was archived as [[etherpadbackup:conftool-T379329]] by [[User:BryanDavis|BryanDavis]] * 02:40 [[etherpad:p/completion-ab-sanity-check]] was archived as [[etherpadbackup:completion-ab-sanity-check]] by [[User:BryanDavis|BryanDavis]] * 02:40 [[etherpad:p/compat-network]] was archived as [[etherpadbackup:compat-network]] by [[User:BryanDavis|BryanDavis]] * 02:39 [[etherpad:p/commons-ios-fixes]] was archived as [[etherpadbackup:commons-ios-fixes]] by [[User:BryanDavis|BryanDavis]] * 02:38 [[etherpad:p/commons-cross-wiki-uploads]] was archived as [[etherpadbackup:commons-cross-wiki-uploads]] by [[User:BryanDavis|BryanDavis]] * 02:38 [[etherpad:p/commons-app-post-draft]] was archived as [[etherpadbackup:commons-app-post-draft]] by [[User:BryanDavis|BryanDavis]] * 02:37 [[etherpad:p/collation]] was archived as [[etherpadbackup:collation]] by [[User:BryanDavis|BryanDavis]] * 02:37 [[etherpad:p/codeacademy-diwanship]] was archived as [[etherpadbackup:codeacademy-diwanship]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/cna4GMpRlJCm65THgk-l]] was archived as [[etherpadbackup:cna4GMpRlJCm65THgk-l]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/cloudvpskernels]] was archived as [[etherpadbackup:cloudvpskernels]] by [[User:BryanDavis|BryanDavis]] * 02:35 [[etherpad:p/cloudvps-no-agent-forwarding]] was archived as [[etherpadbackup:cloudvps-no-agent-forwarding]] by [[User:BryanDavis|BryanDavis]] * 02:35 [[etherpad:p/cloudvirt1018-20190213]] was archived as [[etherpadbackup:cloudvirt1018-20190213]] by [[User:BryanDavis|BryanDavis]] * 02:34 [[etherpad:p/cloudservices1003-upgrade]] was archived as [[etherpadbackup:cloudservices1003-upgrade]] by [[User:BryanDavis|BryanDavis]] * 02:34 [[etherpad:p/cloudreboots]] was archived as [[etherpadbackup:cloudreboots]] by [[User:BryanDavis|BryanDavis]] * 02:33 [[etherpad:p/cloudprojectrequeststweak]] was archived as [[etherpadbackup:cloudprojectrequeststweak]] by [[User:BryanDavis|BryanDavis]] * 02:33 [[etherpad:p/cloudnative]] was archived as [[etherpadbackup:cloudnative]] by [[User:BryanDavis|BryanDavis]] * 02:32 [[etherpad:p/cloudgw-migration]] was archived as [[etherpadbackup:cloudgw-migration]] by [[User:BryanDavis|BryanDavis]] * 02:32 [[etherpad:p/cloudfontcdn]] was archived as [[etherpadbackup:cloudfontcdn]] by [[User:BryanDavis|BryanDavis]] * 02:31 [[etherpad:p/cloudelastic-mystery-red]] was archived as [[etherpadbackup:cloudelastic-mystery-red]] by [[User:BryanDavis|BryanDavis]] * 02:31 [[etherpad:p/cloudelastic-ipip]] was archived as [[etherpadbackup:cloudelastic-ipip]] by [[User:BryanDavis|BryanDavis]] * 02:30 [[etherpad:p/cloudelastic-2-opensearch]] was archived as [[etherpadbackup:cloudelastic-2-opensearch]] by [[User:BryanDavis|BryanDavis]] * 02:30 [[etherpad:p/cloud_purge_2018_ping]] was archived as [[etherpadbackup:cloud_purge_2018_ping]] by [[User:BryanDavis|BryanDavis]] * 02:29 [[etherpad:p/cloud_announcement_20180910]] was archived as [[etherpadbackup:cloud_announcement_20180910]] by [[User:BryanDavis|BryanDavis]] * 02:29 [[etherpad:p/cloud-vps-purge-deletions]] was archived as [[etherpadbackup:cloud-vps-purge-deletions]] by [[User:BryanDavis|BryanDavis]] * 02:28 [[etherpad:p/citoid-TD-fun]] was archived as [[etherpadbackup:citoid-TD-fun]] by [[User:BryanDavis|BryanDavis]] * 02:28 [[etherpad:p/cirrus_cross_project_search_debug_settings]] was archived as [[etherpadbackup:cirrus_cross_project_search_debug_settings]] by [[User:BryanDavis|BryanDavis]] * 02:27 [[etherpad:p/ci-apt-docker-why]] was archived as [[etherpadbackup:ci-apt-docker-why]] by [[User:BryanDavis|BryanDavis]] * 02:27 [[etherpad:p/chrismcmahon]] was archived as [[etherpadbackup:chrismcmahon]] by [[User:BryanDavis|BryanDavis]] * 02:26 [[etherpad:p/changeuid]] was archived as [[etherpadbackup:changeuid]] by [[User:BryanDavis|BryanDavis]] * 02:26 [[etherpad:p/changes]] was archived as [[etherpadbackup:changes]] by [[User:BryanDavis|BryanDavis]] * 02:25 [[etherpad:p/ceph-poc-to-prod]] was archived as [[etherpadbackup:ceph-poc-to-prod]] by [[User:BryanDavis|BryanDavis]] * 02:25 [[etherpad:p/categories-lag]] was archived as [[etherpadbackup:categories-lag]] by [[User:BryanDavis|BryanDavis]] * 02:24 [[etherpad:p/calendar-minimums]] was archived as [[etherpadbackup:calendar-minimums]] by [[User:BryanDavis|BryanDavis]] * 02:24 [[etherpad:p/busterreminder]] was archived as [[etherpadbackup:busterreminder]] by [[User:BryanDavis|BryanDavis]] * 02:23 [[etherpad:p/bug35939]] was archived as [[etherpadbackup:bug35939]] by [[User:BryanDavis|BryanDavis]] * 02:22 [[etherpad:p/brokenpms]] was archived as [[etherpadbackup:brokenpms]] by [[User:BryanDavis|BryanDavis]] * 02:22 [[etherpad:p/bloated-website-code-drains-yo]] was archived as [[etherpadbackup:bloated-website-code-drains-yo]] by [[User:BryanDavis|BryanDavis]] * 02:21 [[etherpad:p/bios-be-gone]] was archived as [[etherpadbackup:bios-be-gone]] by [[User:BryanDavis|BryanDavis]] * 02:21 [[etherpad:p/bigbrother-almost-no-more]] was archived as [[etherpadbackup:bigbrother-almost-no-more]] by [[User:BryanDavis|BryanDavis]] * 02:20 [[etherpad:p/beta-new-ext]] was archived as [[etherpadbackup:beta-new-ext]] by [[User:BryanDavis|BryanDavis]] * 02:20 [[etherpad:p/bernd-onboarding]] was archived as [[etherpadbackup:bernd-onboarding]] by [[User:BryanDavis|BryanDavis]] * 02:19 [[etherpad:p/beginner-hackathons-thoughts]] was archived as [[etherpadbackup:beginner-hackathons-thoughts]] by [[User:BryanDavis|BryanDavis]] * 02:19 [[etherpad:p/bash]] was archived as [[etherpadbackup:bash]] by [[User:BryanDavis|BryanDavis]] * 02:18 [[etherpad:p/bangarang]] was archived as [[etherpadbackup:bangarang]] by [[User:BryanDavis|BryanDavis]] * 02:18 [[etherpad:p/band_names_web]] was archived as [[etherpadbackup:band_names_web]] by [[User:BryanDavis|BryanDavis]] * 02:17 [[etherpad:p/avro]] was archived as [[etherpadbackup:avro]] by [[User:BryanDavis|BryanDavis]] * 02:17 [[etherpad:p/aqs-standup]] was archived as [[etherpadbackup:aqs-standup]] by [[User:BryanDavis|BryanDavis]] * 02:16 [[etherpad:p/app_q3_planning]] was archived as [[etherpadbackup:app_q3_planning]] by [[User:BryanDavis|BryanDavis]] * 02:16 [[etherpad:p/app_q3_health_check]] was archived as [[etherpadbackup:app_q3_health_check]] by [[User:BryanDavis|BryanDavis]] * 02:15 [[etherpad:p/app_process_improvements]] was archived as [[etherpadbackup:app_process_improvements]] by [[User:BryanDavis|BryanDavis]] * 02:15 [[etherpad:p/appQ4planning]] was archived as [[etherpadbackup:appQ4planning]] by [[User:BryanDavis|BryanDavis]] * 02:14 [[etherpad:p/app-strategy-discussion]] was archived as [[etherpadbackup:app-strategy-discussion]] by [[User:BryanDavis|BryanDavis]] * 02:14 [[etherpad:p/app-server-upgrade]] was archived as [[etherpadbackup:app-server-upgrade]] by [[User:BryanDavis|BryanDavis]] * 02:13 [[etherpad:p/app-1-1beta4-3-1beta3-release-notes]] was archived as [[etherpadbackup:app-1-1beta4-3-1beta3-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:13 [[etherpad:p/apiv2]] was archived as [[etherpadbackup:apiv2]] by [[User:BryanDavis|BryanDavis]] * 02:12 [[etherpad:p/api.php]] was archived as [[etherpadbackup:api.php]] by [[User:BryanDavis|BryanDavis]] * 02:12 [[etherpad:p/androidappfeedback]] was archived as [[etherpadbackup:androidappfeedback]] by [[User:BryanDavis|BryanDavis]] * 02:11 [[etherpad:p/android-ime]] was archived as [[etherpadbackup:android-ime]] by [[User:BryanDavis|BryanDavis]] * 02:11 [[etherpad:p/android-commons-app-1-0]] was archived as [[etherpadbackup:android-commons-app-1-0]] by [[User:BryanDavis|BryanDavis]] * 02:10 [[etherpad:p/android-1-1rc1-release-notes]] was archived as [[etherpadbackup:android-1-1rc1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:10 [[etherpad:p/android-1-1beta1-release-notes]] was archived as [[etherpadbackup:android-1-1beta1-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:09 [[etherpad:p/android-1-1-iOS-3-1-third-beta-notes]] was archived as [[etherpadbackup:android-1-1-iOS-3-1-third-beta-notes]] by [[User:BryanDavis|BryanDavis]] * 02:08 [[etherpad:p/android-1-1-beta2-release-notes]] was archived as [[etherpadbackup:android-1-1-beta2-release-notes]] by [[User:BryanDavis|BryanDavis]] * 02:08 [[etherpad:p/analytics163]] was archived as [[etherpadbackup:analytics163]] by [[User:BryanDavis|BryanDavis]] * 02:07 [[etherpad:p/analytics1003-reinstall]] was archived as [[etherpadbackup:analytics1003-reinstall]] by [[User:BryanDavis|BryanDavis]] * 02:07 [[etherpad:p/analytics-todo]] was archived as [[etherpadbackup:analytics-todo]] by [[User:BryanDavis|BryanDavis]] * 02:06 [[etherpad:p/analytics-tech-debt]] was archived as [[etherpadbackup:analytics-tech-debt]] by [[User:BryanDavis|BryanDavis]] * 02:06 [[etherpad:p/analytics-staff-meeting]] was archived as [[etherpadbackup:analytics-staff-meeting]] by [[User:BryanDavis|BryanDavis]] * 02:05 [[etherpad:p/analytics-row-d-maintenance]] was archived as [[etherpadbackup:analytics-row-d-maintenance]] by [[User:BryanDavis|BryanDavis]] * 02:05 [[etherpad:p/analytics-retrospective]] was archived as [[etherpadbackup:analytics-retrospective]] by [[User:BryanDavis|BryanDavis]] * 02:04 [[etherpad:p/analytics-ops-druid]] was archived as [[etherpadbackup:analytics-ops-druid]] by [[User:BryanDavis|BryanDavis]] * 02:04 [[etherpad:p/analytics-ops]] was archived as [[etherpadbackup:analytics-ops]] by [[User:BryanDavis|BryanDavis]] * 02:03 [[etherpad:p/analytics-oozie-13102016]] was archived as [[etherpadbackup:analytics-oozie-13102016]] by [[User:BryanDavis|BryanDavis]] * 02:03 [[etherpad:p/analytics-naming]] was archived as [[etherpadbackup:analytics-naming]] by [[User:BryanDavis|BryanDavis]] * 02:02 [[etherpad:p/analytics-mwds-2015]] was archived as [[etherpadbackup:analytics-mwds-2015]] by [[User:BryanDavis|BryanDavis]] * 02:02 [[etherpad:p/analytics-ingestion]] was archived as [[etherpadbackup:analytics-ingestion]] by [[User:BryanDavis|BryanDavis]] * 02:01 [[etherpad:p/analytics-hive-udf]] was archived as [[etherpadbackup:analytics-hive-udf]] by [[User:BryanDavis|BryanDavis]] * 02:01 [[etherpad:p/analytics-gobblin-mess]] was archived as [[etherpadbackup:analytics-gobblin-mess]] by [[User:BryanDavis|BryanDavis]] * 02:00 [[etherpad:p/analytics-goals]] was archived as [[etherpadbackup:analytics-goals]] by [[User:BryanDavis|BryanDavis]] * 02:00 [[etherpad:p/analytics-eventlogging-postmortem-2014-11]] was archived as [[etherpadbackup:analytics-eventlogging-postmortem-2014-11]] by [[User:BryanDavis|BryanDavis]] * 01:59 [[etherpad:p/analytics-email-drafts]] was archived as [[etherpadbackup:analytics-email-drafts]] by [[User:BryanDavis|BryanDavis]] * 01:59 [[etherpad:p/analytics-druid-upgrade]] was archived as [[etherpadbackup:analytics-druid-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:58 [[etherpad:p/analytics-druid-migration]] was archived as [[etherpadbackup:analytics-druid-migration]] by [[User:BryanDavis|BryanDavis]] * 01:58 [[etherpad:p/analytics-deploy-aqs]] was archived as [[etherpadbackup:analytics-deploy-aqs]] by [[User:BryanDavis|BryanDavis]] * 01:57 [[etherpad:p/analytics-bikeshedding]] was archived as [[etherpadbackup:analytics-bikeshedding]] by [[User:BryanDavis|BryanDavis]] * 01:57 [[etherpad:p/analytics-bikeshed]] was archived as [[etherpadbackup:analytics-bikeshed]] by [[User:BryanDavis|BryanDavis]] * 01:56 [[etherpad:p/analytics-analytics1003-jessie-upgrade]] was archived as [[etherpadbackup:analytics-analytics1003-jessie-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:56 [[etherpad:p/analytics-VE]] was archived as [[etherpadbackup:analytics-VE]] by [[User:BryanDavis|BryanDavis]] * 01:55 [[etherpad:p/analytics-73331]] was archived as [[etherpadbackup:analytics-73331]] by [[User:BryanDavis|BryanDavis]] * 01:55 [[etherpad:p/analytics-72745]] was archived as [[etherpadbackup:analytics-72745]] by [[User:BryanDavis|BryanDavis]] * 01:54 [[etherpad:p/analytics-72740]] was archived as [[etherpadbackup:analytics-72740]] by [[User:BryanDavis|BryanDavis]] * 01:54 [[etherpad:p/analytics-72739]] was archived as [[etherpadbackup:analytics-72739]] by [[User:BryanDavis|BryanDavis]] * 01:53 [[etherpad:p/analytics-72738]] was archived as [[etherpadbackup:analytics-72738]] by [[User:BryanDavis|BryanDavis]] * 01:52 [[etherpad:p/analytics-72737]] was archived as [[etherpadbackup:analytics-72737]] by [[User:BryanDavis|BryanDavis]] * 01:52 [[etherpad:p/analytics-72736]] was archived as [[etherpadbackup:analytics-72736]] by [[User:BryanDavis|BryanDavis]] * 01:51 [[etherpad:p/analytics-72735]] was archived as [[etherpadbackup:analytics-72735]] by [[User:BryanDavis|BryanDavis]] * 01:51 [[etherpad:p/analytics-69254]] was archived as [[etherpadbackup:analytics-69254]] by [[User:BryanDavis|BryanDavis]] * 01:50 [[etherpad:p/alpha-signature-change]] was archived as [[etherpadbackup:alpha-signature-change]] by [[User:BryanDavis|BryanDavis]] * 01:50 [[etherpad:p/ai_vision_wikimedia]] was archived as [[etherpadbackup:ai_vision_wikimedia]] by [[User:BryanDavis|BryanDavis]] * 01:49 [[etherpad:p/aharoni-20160420]] was archived as [[etherpadbackup:aharoni-20160420]] by [[User:BryanDavis|BryanDavis]] * 01:49 [[etherpad:p/aftereffects]] was archived as [[etherpadbackup:aftereffects]] by [[User:BryanDavis|BryanDavis]] * 01:48 [[etherpad:p/admin_accounts_cleanup]] was archived as [[etherpadbackup:admin_accounts_cleanup]] by [[User:BryanDavis|BryanDavis]] * 01:48 [[etherpad:p/admin]] was archived as [[etherpadbackup:admin]] by [[User:BryanDavis|BryanDavis]] * 01:47 [[etherpad:p/addwiki]] was archived as [[etherpadbackup:addwiki]] by [[User:BryanDavis|BryanDavis]] * 01:47 [[etherpad:p/addshore]] was archived as [[etherpadbackup:addshore]] by [[User:BryanDavis|BryanDavis]] * 01:46 [[etherpad:p/actionbar_checklist]] was archived as [[etherpadbackup:actionbar_checklist]] by [[User:BryanDavis|BryanDavis]] * 01:46 [[etherpad:p/access_request_wikis]] was archived as [[etherpadbackup:access_request_wikis]] by [[User:BryanDavis|BryanDavis]] * 01:45 [[etherpad:p/absents]] was archived as [[etherpadbackup:absents]] by [[User:BryanDavis|BryanDavis]] * 01:45 [[etherpad:p/a4NCijsjQp]] was archived as [[etherpadbackup:a4NCijsjQp]] by [[User:BryanDavis|BryanDavis]] * 01:44 [[etherpad:p/ZVFiVE3E09]] was archived as [[etherpadbackup:ZVFiVE3E09]] by [[User:BryanDavis|BryanDavis]] * 01:44 [[etherpad:p/XcC5u59nDdRc_itxINnb]] was archived as [[etherpadbackup:XcC5u59nDdRc_itxINnb]] by [[User:BryanDavis|BryanDavis]] * 01:43 [[etherpad:p/XH6WkSmARmw8dy7UMiIR]] was archived as [[etherpadbackup:XH6WkSmARmw8dy7UMiIR]] by [[User:BryanDavis|BryanDavis]] * 01:43 [[etherpad:p/WsWWUiRllf]] was archived as [[etherpadbackup:WsWWUiRllf]] by [[User:BryanDavis|BryanDavis]] * 01:42 [[etherpad:p/WorT2b8qpR]] was archived as [[etherpadbackup:WorT2b8qpR]] by [[User:BryanDavis|BryanDavis]] * 01:42 [[etherpad:p/Wj4HcOWoHk]] was archived as [[etherpadbackup:Wj4HcOWoHk]] by [[User:BryanDavis|BryanDavis]] * 01:41 [[etherpad:p/WikipediaiOSTesting]] was archived as [[etherpadbackup:WikipediaiOSTesting]] by [[User:BryanDavis|BryanDavis]] * 01:41 [[etherpad:p/WikipediaZeroRetrospective]] was archived as [[etherpadbackup:WikipediaZeroRetrospective]] by [[User:BryanDavis|BryanDavis]] * 01:40 [[etherpad:p/WikipediaPhoneGapAndroidMeetup]] was archived as [[etherpadbackup:WikipediaPhoneGapAndroidMeetup]] by [[User:BryanDavis|BryanDavis]] * 01:40 [[etherpad:p/WikipediaMobileiOS3-1Changelog]] was archived as [[etherpadbackup:WikipediaMobileiOS3-1Changelog]] by [[User:BryanDavis|BryanDavis]] * 01:39 [[etherpad:p/WikipediaMobileAndroidV1-0-1]] was archived as [[etherpadbackup:WikipediaMobileAndroidV1-0-1]] by [[User:BryanDavis|BryanDavis]] * 01:39 [[etherpad:p/WikipediaMobileAndroidRelease]] was archived as [[etherpadbackup:WikipediaMobileAndroidRelease]] by [[User:BryanDavis|BryanDavis]] * 01:38 [[etherpad:p/WikipediaMobileAndroid-V1-0-2]] was archived as [[etherpadbackup:WikipediaMobileAndroid-V1-0-2]] by [[User:BryanDavis|BryanDavis]] * 01:37 [[etherpad:p/WikipediaMobile-3-1-1-scratchpad]] was archived as [[etherpadbackup:WikipediaMobile-3-1-1-scratchpad]] by [[User:BryanDavis|BryanDavis]] * 01:37 [[etherpad:p/WikipediaLite]] was archived as [[etherpadbackup:WikipediaLite]] by [[User:BryanDavis|BryanDavis]] * 01:36 [[etherpad:p/Wikipedia-iOS-Rejection-Message]] was archived as [[etherpadbackup:Wikipedia-iOS-Rejection-Message]] by [[User:BryanDavis|BryanDavis]] * 01:36 [[etherpad:p/Wikipedia-iOS-3-1-1-issues]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1-issues]] by [[User:BryanDavis|BryanDavis]] * 01:35 [[etherpad:p/Wikipedia-iOS-3-1-1]] was archived as [[etherpadbackup:Wikipedia-iOS-3-1-1]] by [[User:BryanDavis|BryanDavis]] * 01:35 [[etherpad:p/Wikipedia-iOS]] was archived as [[etherpadbackup:Wikipedia-iOS]] by [[User:BryanDavis|BryanDavis]] * 01:34 [[etherpad:p/WikipdiaMobileRC4]] was archived as [[etherpadbackup:WikipdiaMobileRC4]] by [[User:BryanDavis|BryanDavis]] * 01:34 [[etherpad:p/WikimediaTelepresence]] was archived as [[etherpadbackup:WikimediaTelepresence]] by [[User:BryanDavis|BryanDavis]] * 01:33 [[etherpad:p/Wikimania_Hackathon_2025_-_Opening_Ceremony]] was archived as [[etherpadbackup:Wikimania_Hackathon_2025_-_Opening_Ceremony]] by [[User:BryanDavis|BryanDavis]] * 01:33 [[etherpad:p/Wikimania_2023_Hackathon]] was archived as [[etherpadbackup:Wikimania_2023_Hackathon]] by [[User:BryanDavis|BryanDavis]] * 01:32 [[etherpad:p/Wikigrok_Q4]] was archived as [[etherpadbackup:Wikigrok_Q4]] by [[User:BryanDavis|BryanDavis]] * 01:32 [[etherpad:p/Wikidata_technical_needs]] was archived as [[etherpadbackup:Wikidata_technical_needs]] by [[User:BryanDavis|BryanDavis]] * 01:31 [[etherpad:p/WikiGrokTest1]] was archived as [[etherpadbackup:WikiGrokTest1]] by [[User:BryanDavis|BryanDavis]] * 01:31 [[etherpad:p/WikiDev16-T114045]] was archived as [[etherpadbackup:WikiDev16-T114045]] by [[User:BryanDavis|BryanDavis]] * 01:30 [[etherpad:p/Watchlist-20beta-20status]] was archived as [[etherpadbackup:Watchlist-20beta-20status]] by [[User:BryanDavis|BryanDavis]] * 01:30 [[etherpad:p/WRN202003]] was archived as [[etherpadbackup:WRN202003]] by [[User:BryanDavis|BryanDavis]] * 01:29 [[etherpad:p/WRN201409]] was archived as [[etherpadbackup:WRN201409]] by [[User:BryanDavis|BryanDavis]] * 01:29 [[etherpad:p/WRN201408]] was archived as [[etherpadbackup:WRN201408]] by [[User:BryanDavis|BryanDavis]] * 01:28 [[etherpad:p/WP_BRD_generally]] was archived as [[etherpadbackup:WP_BRD_generally]] by [[User:BryanDavis|BryanDavis]] * 01:28 [[etherpad:p/WMTC19-T238265]] was archived as [[etherpadbackup:WMTC19-T238265]] by [[User:BryanDavis|BryanDavis]] * 01:27 [[etherpad:p/WMTC19-T238227]] was archived as [[etherpadbackup:WMTC19-T238227]] by [[User:BryanDavis|BryanDavis]] * 01:27 [[etherpad:p/WMTC19-T234662]] was archived as [[etherpadbackup:WMTC19-T234662]] by [[User:BryanDavis|BryanDavis]] * 01:26 [[etherpad:p/WMTC19-T234655]] was archived as [[etherpadbackup:WMTC19-T234655]] by [[User:BryanDavis|BryanDavis]] * 01:26 [[etherpad:p/WMTC19-T234654]] was archived as [[etherpadbackup:WMTC19-T234654]] by [[User:BryanDavis|BryanDavis]] * 01:25 [[etherpad:p/WMTC19-T234649]] was archived as [[etherpadbackup:WMTC19-T234649]] by [[User:BryanDavis|BryanDavis]] * 01:25 [[etherpad:p/WMTC19-T234641]] was archived as [[etherpadbackup:WMTC19-T234641]] by [[User:BryanDavis|BryanDavis]] * 01:24 [[etherpad:p/WMTC19-T234636]] was archived as [[etherpadbackup:WMTC19-T234636]] by [[User:BryanDavis|BryanDavis]] * 01:24 [[etherpad:p/WMTC19-T234632]] was archived as [[etherpadbackup:WMTC19-T234632]] by [[User:BryanDavis|BryanDavis]] * 01:23 [[etherpad:p/WMHack_Feature_Ideas_for_WorkAdventure]] was archived as [[etherpadbackup:WMHack_Feature_Ideas_for_WorkAdventure]] by [[User:BryanDavis|BryanDavis]] * 01:23 [[etherpad:p/WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] was archived as [[etherpadbackup:WMHack25__Wikimedia_Hackathon_y_Latinoamérica]] by [[User:BryanDavis|BryanDavis]] * 01:22 [[etherpad:p/WMF_Research_Office_Hours|Notes]] was archived as [[etherpadbackup:WMF_Research_Office_Hours|Notes]] by [[User:BryanDavis|BryanDavis]] * 01:21 [[etherpad:p/WMCS-techsupport-task]] was archived as [[etherpadbackup:WMCS-techsupport-task]] by [[User:BryanDavis|BryanDavis]] * 01:21 [[etherpad:p/WMCS-infrafoundations-2021-11-24]] was archived as [[etherpadbackup:WMCS-infrafoundations-2021-11-24]] by [[User:BryanDavis|BryanDavis]] * 01:20 [[etherpad:p/WMCS-Renaming-Announce]] was archived as [[etherpadbackup:WMCS-Renaming-Announce]] by [[User:BryanDavis|BryanDavis]] * 01:20 [[etherpad:p/WMCS-PDU-ops-2019-10-24]] was archived as [[etherpadbackup:WMCS-PDU-ops-2019-10-24]] by [[User:BryanDavis|BryanDavis]] * 01:19 [[etherpad:p/WMCS-2026-02-12]] was archived as [[etherpadbackup:WMCS-2026-02-12]] by [[User:BryanDavis|BryanDavis]] * 01:19 [[etherpad:p/WMCS-2025-12-18]] was archived as [[etherpadbackup:WMCS-2025-12-18]] by [[User:BryanDavis|BryanDavis]] * 01:18 [[etherpad:p/WMCS-2025-09-04]] was archived as [[etherpadbackup:WMCS-2025-09-04]] by [[User:BryanDavis|BryanDavis]] * 01:18 [[etherpad:p/WMCS-2025-08-14]] was archived as [[etherpadbackup:WMCS-2025-08-14]] by [[User:BryanDavis|BryanDavis]] * 01:17 [[etherpad:p/WMCS-2024-02-07]] was archived as [[etherpadbackup:WMCS-2024-02-07]] by [[User:BryanDavis|BryanDavis]] * 01:17 [[etherpad:p/WMCS-2023-10-04]] was archived as [[etherpadbackup:WMCS-2023-10-04]] by [[User:BryanDavis|BryanDavis]] * 01:16 [[etherpad:p/WMCS-2023-09-20]] was archived as [[etherpadbackup:WMCS-2023-09-20]] by [[User:BryanDavis|BryanDavis]] * 01:16 [[etherpad:p/WMCS-2023-08-10]] was archived as [[etherpadbackup:WMCS-2023-08-10]] by [[User:BryanDavis|BryanDavis]] * 01:15 [[etherpad:p/WMCS-2023-06-28]] was archived as [[etherpadbackup:WMCS-2023-06-28]] by [[User:BryanDavis|BryanDavis]] * 01:15 [[etherpad:p/WMCS-2023-04-09-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2023-04-09-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:14 [[etherpad:p/WMCS-2022-11-09]] was archived as [[etherpadbackup:WMCS-2022-11-09]] by [[User:BryanDavis|BryanDavis]] * 01:14 [[etherpad:p/WMCS-2022-09-07]] was archived as [[etherpadbackup:WMCS-2022-09-07]] by [[User:BryanDavis|BryanDavis]] * 01:13 [[etherpad:p/WMCS-2022-07-20]] was archived as [[etherpadbackup:WMCS-2022-07-20]] by [[User:BryanDavis|BryanDavis]] * 01:13 [[etherpad:p/WMCS-2022-07-13]] was archived as [[etherpadbackup:WMCS-2022-07-13]] by [[User:BryanDavis|BryanDavis]] * 01:12 [[etherpad:p/WMCS-2022-05-11-tools-k8s-upgrade]] was archived as [[etherpadbackup:WMCS-2022-05-11-tools-k8s-upgrade]] by [[User:BryanDavis|BryanDavis]] * 01:12 [[etherpad:p/WMCS-2021-12-15]] was archived as [[etherpadbackup:WMCS-2021-12-15]] by [[User:BryanDavis|BryanDavis]] * 01:11 [[etherpad:p/WMCS-2021-11-03]] was archived as [[etherpadbackup:WMCS-2021-11-03]] by [[User:BryanDavis|BryanDavis]] * 01:10 [[etherpad:p/WMCS-2020-12-02]] was archived as [[etherpadbackup:WMCS-2020-12-02]] by [[User:BryanDavis|BryanDavis]] * 01:10 [[etherpad:p/WMCS-2020-11-25]] was archived as [[etherpadbackup:WMCS-2020-11-25]] by [[User:BryanDavis|BryanDavis]] * 01:09 [[etherpad:p/WMCS-2020-11-04]] was archived as [[etherpadbackup:WMCS-2020-11-04]] by [[User:BryanDavis|BryanDavis]] * 01:09 [[etherpad:p/WMCS-2020-04-22]] was archived as [[etherpadbackup:WMCS-2020-04-22]] by [[User:BryanDavis|BryanDavis]] * 01:08 [[etherpad:p/WMCS-2020-01-07]] was archived as [[etherpadbackup:WMCS-2020-01-07]] by [[User:BryanDavis|BryanDavis]] * 01:08 [[etherpad:p/WMCS-2019-10-24]] was archived as [[etherpadbackup:WMCS-2019-10-24]] by [[User:BryanDavis|BryanDavis]] * 01:07 [[etherpad:p/WMCS-2019-09-03]] was archived as [[etherpadbackup:WMCS-2019-09-03]] by [[User:BryanDavis|BryanDavis]] * 01:07 [[etherpad:p/WMCS-2019-07-30]] was archived as [[etherpadbackup:WMCS-2019-07-30]] by [[User:BryanDavis|BryanDavis]] * 01:06 [[etherpad:p/WMCS-2019-01-29]] was archived as [[etherpadbackup:WMCS-2019-01-29]] by [[User:BryanDavis|BryanDavis]] * 01:06 [[etherpad:p/WMCS-2018-10-02]] was archived as [[etherpadbackup:WMCS-2018-10-02]] by [[User:BryanDavis|BryanDavis]] * 01:05 [[etherpad:p/WMCS-2018-08-14]] was archived as [[etherpadbackup:WMCS-2018-08-14]] by [[User:BryanDavis|BryanDavis]] * 01:05 [[etherpad:p/WMCS-2018-07-31]] was archived as [[etherpadbackup:WMCS-2018-07-31]] by [[User:BryanDavis|BryanDavis]] * 01:04 [[etherpad:p/WMCS-2018-07-10]] was archived as [[etherpadbackup:WMCS-2018-07-10]] by [[User:BryanDavis|BryanDavis]] * 01:04 [[etherpad:p/WMCS-2018-06-19]] was archived as [[etherpadbackup:WMCS-2018-06-19]] by [[User:BryanDavis|BryanDavis]] * 01:03 [[etherpad:p/WMCS-2018-06-05]] was archived as [[etherpadbackup:WMCS-2018-06-05]] by [[User:BryanDavis|BryanDavis]] * 01:03 [[etherpad:p/WMCS-2018-05-29]] was archived as [[etherpadbackup:WMCS-2018-05-29]] by [[User:BryanDavis|BryanDavis]] * 01:02 [[etherpad:p/WMCS-2018-05-01]] was archived as [[etherpadbackup:WMCS-2018-05-01]] by [[User:BryanDavis|BryanDavis]] * 01:02 [[etherpad:p/WMCS-2018-04-10]] was archived as [[etherpadbackup:WMCS-2018-04-10]] by [[User:BryanDavis|BryanDavis]] * 01:01 [[etherpad:p/WMCS-2018-04-03]] was archived as [[etherpadbackup:WMCS-2018-04-03]] by [[User:BryanDavis|BryanDavis]] * 01:01 [[etherpad:p/WMCS-2018-02-13]] was archived as [[etherpadbackup:WMCS-2018-02-13]] by [[User:BryanDavis|BryanDavis]] * 01:00 [[etherpad:p/WMCS-2018-02-05]] was archived as [[etherpadbackup:WMCS-2018-02-05]] by [[User:BryanDavis|BryanDavis]] * 01:00 [[etherpad:p/WMCS-2018-01-23]] was archived as [[etherpadbackup:WMCS-2018-01-23]] by [[User:BryanDavis|BryanDavis]] * 00:59 [[etherpad:p/WMCS-2018-01-09]] was archived as [[etherpadbackup:WMCS-2018-01-09]] by [[User:BryanDavis|BryanDavis]] * 00:59 [[etherpad:p/WMCS-2017-12-19]] was archived as [[etherpadbackup:WMCS-2017-12-19]] by [[User:BryanDavis|BryanDavis]] * 00:58 [[etherpad:p/WMCS-2017-11-28]] was archived as [[etherpadbackup:WMCS-2017-11-28]] by [[User:BryanDavis|BryanDavis]] * 00:58 [[etherpad:p/WMCS-2017-11-21]] was archived as [[etherpadbackup:WMCS-2017-11-21]] by [[User:BryanDavis|BryanDavis]] * 00:57 [[etherpad:p/WMCS-2017-11-14]] was archived as [[etherpadbackup:WMCS-2017-11-14]] by [[User:BryanDavis|BryanDavis]] * 00:57 [[etherpad:p/WMCS-2017-11-07]] was archived as [[etherpadbackup:WMCS-2017-11-07]] by [[User:BryanDavis|BryanDavis]] * 00:56 [[etherpad:p/WMCS-2017-10-30]] was archived as [[etherpadbackup:WMCS-2017-10-30]] by [[User:BryanDavis|BryanDavis]] * 00:56 [[etherpad:p/WMCS-2017-10-24]] was archived as [[etherpadbackup:WMCS-2017-10-24]] by [[User:BryanDavis|BryanDavis]] * 00:55 [[etherpad:p/WMCS-2017-10-03]] was archived as [[etherpadbackup:WMCS-2017-10-03]] by [[User:BryanDavis|BryanDavis]] * 00:54 [[etherpad:p/WMCS-2017-09-19]] was archived as [[etherpadbackup:WMCS-2017-09-19]] by [[User:BryanDavis|BryanDavis]] * 00:54 [[etherpad:p/WMCS-2017-05-15]] was archived as [[etherpadbackup:WMCS-2017-05-15]] by [[User:BryanDavis|BryanDavis]] * 00:53 [[etherpad:p/WMCS-2017-05-08]] was archived as [[etherpadbackup:WMCS-2017-05-08]] by [[User:BryanDavis|BryanDavis]] * 00:53 [[etherpad:p/WMCS-2017-04-17]] was archived as [[etherpadbackup:WMCS-2017-04-17]] by [[User:BryanDavis|BryanDavis]] * 00:52 [[etherpad:p/WMCS-2017-04-10]] was archived as [[etherpadbackup:WMCS-2017-04-10]] by [[User:BryanDavis|BryanDavis]] * 00:52 [[etherpad:p/WM2024_Day4_Lviv-_Rooms_21+22+23]] was archived as [[etherpadbackup:WM2024_Day4_Lviv-_Rooms_21+22+23]] by [[User:BryanDavis|BryanDavis]] * 00:51 [[etherpad:p/WM2024_Day1_Warsaw-_Rooms_20+24]] was archived as [[etherpadbackup:WM2024_Day1_Warsaw-_Rooms_20+24]] by [[User:BryanDavis|BryanDavis]] * 00:51 [[etherpad:p/WLMshowcase]] was archived as [[etherpadbackup:WLMshowcase]] by [[User:BryanDavis|BryanDavis]] * 00:50 [[etherpad:p/WLMShowcase2]] was archived as [[etherpadbackup:WLMShowcase2]] by [[User:BryanDavis|BryanDavis]] * 00:50 [[etherpad:p/WLMMobile]] was archived as [[etherpadbackup:WLMMobile]] by [[User:BryanDavis|BryanDavis]] * 00:49 [[etherpad:p/WLMMarketToDo]] was archived as [[etherpadbackup:WLMMarketToDo]] by [[User:BryanDavis|BryanDavis]] * 00:49 [[etherpad:p/WLMAppEmail]] was archived as [[etherpadbackup:WLMAppEmail]] by [[User:BryanDavis|BryanDavis]] * 00:48 [[etherpad:p/WLM-on-cluster]] was archived as [[etherpadbackup:WLM-on-cluster]] by [[User:BryanDavis|BryanDavis]] * 00:48 [[etherpad:p/WLM-mobile]] was archived as [[etherpadbackup:WLM-mobile]] by [[User:BryanDavis|BryanDavis]] * 00:47 [[etherpad:p/WLM-API-to-WMF]] was archived as [[etherpadbackup:WLM-API-to-WMF]] by [[User:BryanDavis|BryanDavis]] * 00:47 [[etherpad:p/WLM-2013-07-05]] was archived as [[etherpadbackup:WLM-2013-07-05]] by [[User:BryanDavis|BryanDavis]] * 00:46 [[etherpad:p/VzkZSy0PKK]] was archived as [[etherpadbackup:VzkZSy0PKK]] by [[User:BryanDavis|BryanDavis]] * 00:46 [[etherpad:p/VisualEditorTeamLunchOrder20140327]] was archived as [[etherpadbackup:VisualEditorTeamLunchOrder20140327]] by [[User:BryanDavis|BryanDavis]] * 00:45 [[etherpad:p/VelocityNYC2015-Day1]] was archived as [[etherpadbackup:VelocityNYC2015-Day1]] by [[User:BryanDavis|BryanDavis]] * 00:45 [[etherpad:p/Vagrant-wannabees]] was archived as [[etherpadbackup:Vagrant-wannabees]] by [[User:BryanDavis|BryanDavis]] * 00:44 [[etherpad:p/VPT]] was archived as [[etherpadbackup:VPT]] by [[User:BryanDavis|BryanDavis]] * 00:44 [[etherpad:p/VM_puppet_failures]] was archived as [[etherpadbackup:VM_puppet_failures]] by [[User:BryanDavis|BryanDavis]] * 00:43 [[etherpad:p/VM_puppet_disabled]] was archived as [[etherpadbackup:VM_puppet_disabled]] by [[User:BryanDavis|BryanDavis]] * 00:43 [[etherpad:p/VE_bugs]] was archived as [[etherpadbackup:VE_bugs]] by [[User:BryanDavis|BryanDavis]] * 00:42 [[etherpad:p/VETOC]] was archived as [[etherpadbackup:VETOC]] by [[User:BryanDavis|BryanDavis]] * 00:42 [[etherpad:p/VEQ3Perf]] was archived as [[etherpadbackup:VEQ3Perf]] by [[User:BryanDavis|BryanDavis]] * 00:41 [[etherpad:p/VE-Chromium-Upstream]] was archived as [[etherpadbackup:VE-Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]] * 00:41 [[etherpad:p/UwSprintAugust2012]] was archived as [[etherpadbackup:UwSprintAugust2012]] by [[User:BryanDavis|BryanDavis]] * 00:40 [[etherpad:p/Upload_debugging]] was archived as [[etherpadbackup:Upload_debugging]] by [[User:BryanDavis|BryanDavis]] * 00:39 [[etherpad:p/UploadWizard-and-EventLogging]] was archived as [[etherpadbackup:UploadWizard-and-EventLogging]] by [[User:BryanDavis|BryanDavis]] * 00:39 [[etherpad:p/Upgrade-now-bitches]] was archived as [[etherpadbackup:Upgrade-now-bitches]] by [[User:BryanDavis|BryanDavis]] * 00:38 [[etherpad:p/UniqueServiceGroups]] was archived as [[etherpadbackup:UniqueServiceGroups]] by [[User:BryanDavis|BryanDavis]] * 00:38 [[etherpad:p/Udeme_Emmanson]] was archived as [[etherpadbackup:Udeme_Emmanson]] by [[User:BryanDavis|BryanDavis]] * 00:37 [[etherpad:p/UShQLRyUMv]] was archived as [[etherpadbackup:UShQLRyUMv]] by [[User:BryanDavis|BryanDavis]] * 00:37 [[etherpad:p/UA-SchemaUpdate]] was archived as [[etherpadbackup:UA-SchemaUpdate]] by [[User:BryanDavis|BryanDavis]] * 00:36 [[etherpad:p/U84-IdW1WfD2vZ4ogSCx]] was archived as [[etherpadbackup:U84-IdW1WfD2vZ4ogSCx]] by [[User:BryanDavis|BryanDavis]] * 00:36 [[etherpad:p/U53g0zdCtb]] was archived as [[etherpadbackup:U53g0zdCtb]] by [[User:BryanDavis|BryanDavis]] * 00:35 [[etherpad:p/U]] was archived as [[etherpadbackup:U]] by [[User:BryanDavis|BryanDavis]] * 00:35 [[etherpad:p/Trusty_grid_shutdown]] was archived as [[etherpadbackup:Trusty_grid_shutdown]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/TransclusionModelIntroduction]] was archived as [[etherpadbackup:TransclusionModelIntroduction]] by [[User:BryanDavis|BryanDavis]] * 00:34 [[etherpad:p/Traffic-2021-08-24]] was archived as [[etherpadbackup:Traffic-2021-08-24]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/Traffic-2021-08-17]] was archived as [[etherpadbackup:Traffic-2021-08-17]] by [[User:BryanDavis|BryanDavis]] * 00:33 [[etherpad:p/Traffic-2021-02-17]] was archived as [[etherpadbackup:Traffic-2021-02-17]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/Traffic-2020-12-16]] was archived as [[etherpadbackup:Traffic-2020-12-16]] by [[User:BryanDavis|BryanDavis]] * 00:32 [[etherpad:p/Traffic-2020-11-02]] was archived as [[etherpadbackup:Traffic-2020-11-02]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/Traffic-2020-10-13]] was archived as [[etherpadbackup:Traffic-2020-10-13]] by [[User:BryanDavis|BryanDavis]] * 00:31 [[etherpad:p/Traffic-2020-09-28]] was archived as [[etherpadbackup:Traffic-2020-09-28]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/Traffic-2018-06-28]] was archived as [[etherpadbackup:Traffic-2018-06-28]] by [[User:BryanDavis|BryanDavis]] * 00:30 [[etherpad:p/Traffic-2018-06-14]] was archived as [[etherpadbackup:Traffic-2018-06-14]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/ToolLabs_Survey_feedback]] was archived as [[etherpadbackup:ToolLabs_Survey_feedback]] by [[User:BryanDavis|BryanDavis]] * 00:29 [[etherpad:p/TextInputWidgetValidation]] was archived as [[etherpadbackup:TextInputWidgetValidation]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/Templates]] was archived as [[etherpadbackup:Templates]] by [[User:BryanDavis|BryanDavis]] * 00:28 [[etherpad:p/Technical_Engagment_All_Hands_2019doing_and_should]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019doing_and_should]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/Technical_Engagment_All_Hands_2019]] was archived as [[etherpadbackup:Technical_Engagment_All_Hands_2019]] by [[User:BryanDavis|BryanDavis]] * 00:27 [[etherpad:p/TechOps-goals-Q4-FY15-16]] was archived as [[etherpadbackup:TechOps-goals-Q4-FY15-16]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/TechOps-goals-FQ3-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ3-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:26 [[etherpad:p/TechOps-goals-FQ2-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ2-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/TechOps-goals-FQ1-FY1617]] was archived as [[etherpadbackup:TechOps-goals-FQ1-FY1617]] by [[User:BryanDavis|BryanDavis]] * 00:25 [[etherpad:p/TechOps-2017-09-18]] was archived as [[etherpadbackup:TechOps-2017-09-18]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/TechOps-2015-10-14]] was archived as [[etherpadbackup:TechOps-2015-10-14]] by [[User:BryanDavis|BryanDavis]] * 00:24 [[etherpad:p/TechOps-2015-07-27]] was archived as [[etherpadbackup:TechOps-2015-07-27]] by [[User:BryanDavis|BryanDavis]] * 00:23 [[etherpad:p/TechDaysCareers]] was archived as [[etherpadbackup:TechDaysCareers]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/TechDays-Mobile]] was archived as [[etherpadbackup:TechDays-Mobile]] by [[User:BryanDavis|BryanDavis]] * 00:22 [[etherpad:p/TargetLoader]] was archived as [[etherpadbackup:TargetLoader]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/T_movie]] was archived as [[etherpadbackup:T_movie]] by [[User:BryanDavis|BryanDavis]] * 00:21 [[etherpad:p/TFMQF]] was archived as [[etherpadbackup:TFMQF]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/T404584-announcement]] was archived as [[etherpadbackup:T404584-announcement]] by [[User:BryanDavis|BryanDavis]] * 00:20 [[etherpad:p/T395855]] was archived as [[etherpadbackup:T395855]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/T372912]] was archived as [[etherpadbackup:T372912]] by [[User:BryanDavis|BryanDavis]] * 00:19 [[etherpad:p/T364459]] was archived as [[etherpadbackup:T364459]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/T340241-new-fingerprints]] was archived as [[etherpadbackup:T340241-new-fingerprints]] by [[User:BryanDavis|BryanDavis]] * 00:18 [[etherpad:p/T340241-new-bastion]] was archived as [[etherpadbackup:T340241-new-bastion]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/T221119_-_log]] was archived as [[etherpadbackup:T221119_-_log]] by [[User:BryanDavis|BryanDavis]] * 00:17 [[etherpad:p/T191532]] was archived as [[etherpadbackup:T191532]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/T182722-outage]] was archived as [[etherpadbackup:T182722-outage]] by [[User:BryanDavis|BryanDavis]] * 00:16 [[etherpad:p/T152122_notes]] was archived as [[etherpadbackup:T152122_notes]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/T128190]] was archived as [[etherpadbackup:T128190]] by [[User:BryanDavis|BryanDavis]] * 00:15 [[etherpad:p/Swift-Switch-Originals]] was archived as [[etherpadbackup:Swift-Switch-Originals]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/Summary/_Hypothesis]] was archived as [[etherpadbackup:Summary/_Hypothesis]] by [[User:BryanDavis|BryanDavis]] * 00:14 [[etherpad:p/SquidToVarnish]] was archived as [[etherpadbackup:SquidToVarnish]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/Sqoop]] was archived as [[etherpadbackup:Sqoop]] by [[User:BryanDavis|BryanDavis]] * 00:13 [[etherpad:p/Sprint_D]] was archived as [[etherpadbackup:Sprint_D]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/ShutdownPreciseGrid]] was archived as [[etherpadbackup:ShutdownPreciseGrid]] by [[User:BryanDavis|BryanDavis]] * 00:12 [[etherpad:p/ServiceOps-AugustReboots]] was archived as [[etherpadbackup:ServiceOps-AugustReboots]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/ServiceOps]] was archived as [[etherpadbackup:ServiceOps]] by [[User:BryanDavis|BryanDavis]] * 00:11 [[etherpad:p/Searchfdbksurvey]] was archived as [[etherpadbackup:Searchfdbksurvey]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/Search_UserStories]] was archived as [[etherpadbackup:Search_UserStories]] by [[User:BryanDavis|BryanDavis]] * 00:10 [[etherpad:p/ScrummasterMeetup-2013-10-10]] was archived as [[etherpadbackup:ScrummasterMeetup-2013-10-10]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/Scrum-of-Scrums-copypaste]] was archived as [[etherpadbackup:Scrum-of-Scrums-copypaste]] by [[User:BryanDavis|BryanDavis]] * 00:09 [[etherpad:p/SandboxHatesMe]] was archived as [[etherpadbackup:SandboxHatesMe]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/SWAT_2016-04-14]] was archived as [[etherpadbackup:SWAT_2016-04-14]] by [[User:BryanDavis|BryanDavis]] * 00:08 [[etherpad:p/SWAT]] was archived as [[etherpadbackup:SWAT]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/SVImxzLIyw]] was archived as [[etherpadbackup:SVImxzLIyw]] by [[User:BryanDavis|BryanDavis]] * 00:07 [[etherpad:p/SRE-goals-FQ4-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ4-FY1819]] by [[User:BryanDavis|BryanDavis]] * 00:06 [[etherpad:p/SRE-goals-FQ3-FY1819]] was archived as [[etherpadbackup:SRE-goals-FQ3-FY1819]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/SRE-goals-FQ1-FY1920]] was archived as [[etherpadbackup:SRE-goals-FQ1-FY1920]] by [[User:BryanDavis|BryanDavis]] * 00:05 [[etherpad:p/SRE-ServiceOps-StatusMeeting]] was archived as [[etherpadbackup:SRE-ServiceOps-StatusMeeting]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/SRE-ServiceOps-2019-01-03]] was archived as [[etherpadbackup:SRE-ServiceOps-2019-01-03]] by [[User:BryanDavis|BryanDavis]] * 00:04 [[etherpad:p/SRE-Foundations-2021-10-25]] was archived as [[etherpadbackup:SRE-Foundations-2021-10-25]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/SRE-Foundations-2020-07-01]] was archived as [[etherpadbackup:SRE-Foundations-2020-07-01]] by [[User:BryanDavis|BryanDavis]] * 00:03 [[etherpad:p/SRE-Foundations-2020-03-11]] was archived as [[etherpadbackup:SRE-Foundations-2020-03-11]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/SRE-Foundations-2020-02-12]] was archived as [[etherpadbackup:SRE-Foundations-2020-02-12]] by [[User:BryanDavis|BryanDavis]] * 00:02 [[etherpad:p/SRE-Foundations-2019-11-13]] was archived as [[etherpadbackup:SRE-Foundations-2019-11-13]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/SRE-Foundations-2019-08-07]] was archived as [[etherpadbackup:SRE-Foundations-2019-08-07]] by [[User:BryanDavis|BryanDavis]] * 00:01 [[etherpad:p/SRE-Foundations-2019-07-31]] was archived as [[etherpadbackup:SRE-Foundations-2019-07-31]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/SOAAuth]] was archived as [[etherpadbackup:SOAAuth]] by [[User:BryanDavis|BryanDavis]] * 00:00 [[etherpad:p/SGE_rebuild]] was archived as [[etherpadbackup:SGE_rebuild]] by [[User:BryanDavis|BryanDavis]] === 2026-05-05 === * 23:59 [[etherpad:p/S]] was archived as [[etherpadbackup:S]] by [[User:BryanDavis|BryanDavis]] * 23:59 [[etherpad:p/RobotsMetaTag]] was archived as [[etherpadbackup:RobotsMetaTag]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/ResourceLoader-for-mobile]] was archived as [[etherpadbackup:ResourceLoader-for-mobile]] by [[User:BryanDavis|BryanDavis]] * 23:58 [[etherpad:p/ResourceLoader-Mobile]] was archived as [[etherpadbackup:ResourceLoader-Mobile]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/ResourceLoader]] was archived as [[etherpadbackup:ResourceLoader]] by [[User:BryanDavis|BryanDavis]] * 23:57 [[etherpad:p/ReleaseNotes]] was archived as [[etherpadbackup:ReleaseNotes]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/ReferenceInspector]] was archived as [[etherpadbackup:ReferenceInspector]] by [[User:BryanDavis|BryanDavis]] * 23:56 [[etherpad:p/Reading_web_sprint_2_names]] was archived as [[etherpadbackup:Reading_web_sprint_2_names]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/ReadingWebQ3Planning]] was archived as [[etherpadbackup:ReadingWebQ3Planning]] by [[User:BryanDavis|BryanDavis]] * 23:55 [[etherpad:p/ReadingWebCodeReviewSessions]] was archived as [[etherpadbackup:ReadingWebCodeReviewSessions]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/ReadingShowcase]] was archived as [[etherpadbackup:ReadingShowcase]] by [[User:BryanDavis|BryanDavis]] * 23:54 [[etherpad:p/Reading]] was archived as [[etherpadbackup:Reading]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/RdvI3eLIj5AMM4PN_hHz]] was archived as [[etherpadbackup:RdvI3eLIj5AMM4PN_hHz]] by [[User:BryanDavis|BryanDavis]] * 23:53 [[etherpad:p/RWW-WikipediaZero]] was archived as [[etherpadbackup:RWW-WikipediaZero]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/RLRLRL]] was archived as [[etherpadbackup:RLRLRL]] by [[User:BryanDavis|BryanDavis]] * 23:52 [[etherpad:p/RLFuncDeps]] was archived as [[etherpadbackup:RLFuncDeps]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/RLExceptions]] was archived as [[etherpadbackup:RLExceptions]] by [[User:BryanDavis|BryanDavis]] * 23:51 [[etherpad:p/RL-mwloader-testcases-tmp]] was archived as [[etherpadbackup:RL-mwloader-testcases-tmp]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/RL-mwloader-testcases]] was archived as [[etherpadbackup:RL-mwloader-testcases]] by [[User:BryanDavis|BryanDavis]] * 23:50 [[etherpad:p/RL-async-head]] was archived as [[etherpadbackup:RL-async-head]] by [[User:BryanDavis|BryanDavis]] * 23:49 [[etherpad:p/RD201412]] was archived as [[etherpadbackup:RD201412]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/RD201411]] was archived as [[etherpadbackup:RD201411]] by [[User:BryanDavis|BryanDavis]] * 23:48 [[etherpad:p/RD201409]] was archived as [[etherpadbackup:RD201409]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/RD2014]] was archived as [[etherpadbackup:RD2014]] by [[User:BryanDavis|BryanDavis]] * 23:47 [[etherpad:p/Quarterly_metrics_scorecard]] was archived as [[etherpadbackup:Quarterly_metrics_scorecard]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/QgiZvx5mWMQzlM03zCC1]] was archived as [[etherpadbackup:QgiZvx5mWMQzlM03zCC1]] by [[User:BryanDavis|BryanDavis]] * 23:46 [[etherpad:p/QR1]] was archived as [[etherpadbackup:QR1]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/QPP]] was archived as [[etherpadbackup:QPP]] by [[User:BryanDavis|BryanDavis]] * 23:45 [[etherpad:p/QA_Automation_sync_up]] was archived as [[etherpadbackup:QA_Automation_sync_up]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/Q]] was archived as [[etherpadbackup:Q]] by [[User:BryanDavis|BryanDavis]] * 23:44 [[etherpad:p/Puppet3]] was archived as [[etherpadbackup:Puppet3]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/ProofreadPageLua]] was archived as [[etherpadbackup:ProofreadPageLua]] by [[User:BryanDavis|BryanDavis]] * 23:43 [[etherpad:p/Phragile-requirements-2015-05]] was archived as [[etherpadbackup:Phragile-requirements-2015-05]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/Phabricator_Upgrade_Planning_20190123]] was archived as [[etherpadbackup:Phabricator_Upgrade_Planning_20190123]] by [[User:BryanDavis|BryanDavis]] * 23:42 [[etherpad:p/PhabricatorInstall]] was archived as [[etherpadbackup:PhabricatorInstall]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/PerformanceUpdate201601]] was archived as [[etherpadbackup:PerformanceUpdate201601]] by [[User:BryanDavis|BryanDavis]] * 23:41 [[etherpad:p/PerformanceDecemberUpdate]] was archived as [[etherpadbackup:PerformanceDecemberUpdate]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/Performance-2015-12-14]] was archived as [[etherpadbackup:Performance-2015-12-14]] by [[User:BryanDavis|BryanDavis]] * 23:40 [[etherpad:p/Performance]] was archived as [[etherpadbackup:Performance]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/PartnerEngineerTraining]] was archived as [[etherpadbackup:PartnerEngineerTraining]] by [[User:BryanDavis|BryanDavis]] * 23:39 [[etherpad:p/PV_Analytics_Infra]] was archived as [[etherpadbackup:PV_Analytics_Infra]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/PVTransition]] was archived as [[etherpadbackup:PVTransition]] by [[User:BryanDavis|BryanDavis]] * 23:38 [[etherpad:p/PAWS-2020-04-28]] was archived as [[etherpadbackup:PAWS-2020-04-28]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/P&EDashboard20160112]] was archived as [[etherpadbackup:P&EDashboard20160112]] by [[User:BryanDavis|BryanDavis]] * 23:37 [[etherpad:p/P]] was archived as [[etherpadbackup:P]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/OsJfzQH80T]] was archived as [[etherpadbackup:OsJfzQH80T]] by [[User:BryanDavis|BryanDavis]] * 23:36 [[etherpad:p/OpsSummit2013Day2]] was archived as [[etherpadbackup:OpsSummit2013Day2]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/OldWmfBranchCleanup]] was archived as [[etherpadbackup:OldWmfBranchCleanup]] by [[User:BryanDavis|BryanDavis]] * 23:35 [[etherpad:p/OctoberMetricsAgenda]] was archived as [[etherpadbackup:OctoberMetricsAgenda]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/OSM-Wikimedia]] was archived as [[etherpadbackup:OSM-Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 23:34 [[etherpad:p/OOjsUI-TODO]] was archived as [[etherpadbackup:OOjsUI-TODO]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/NewMobileTech]] was archived as [[etherpadbackup:NewMobileTech]] by [[User:BryanDavis|BryanDavis]] * 23:33 [[etherpad:p/NewDeploysToday]] was archived as [[etherpadbackup:NewDeploysToday]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/N_sprint]] was archived as [[etherpadbackup:N_sprint]] by [[User:BryanDavis|BryanDavis]] * 23:32 [[etherpad:p/NQpvqJeiKg]] was archived as [[etherpadbackup:NQpvqJeiKg]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/N5q9EmRswClmLQyBOezv]] was archived as [[etherpadbackup:N5q9EmRswClmLQyBOezv]] by [[User:BryanDavis|BryanDavis]] * 23:31 [[etherpad:p/Mobile_at_Wikimania_2015]] was archived as [[etherpadbackup:Mobile_at_Wikimania_2015]] by [[User:BryanDavis|BryanDavis]] * 23:30 [[etherpad:p/Mobile_Parsoid]] was archived as [[etherpadbackup:Mobile_Parsoid]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/MobileWeb_Iteration_name]] was archived as [[etherpadbackup:MobileWeb_Iteration_name]] by [[User:BryanDavis|BryanDavis]] * 23:29 [[etherpad:p/MobileWebOnboarding]] was archived as [[etherpadbackup:MobileWebOnboarding]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/MobileWebDevelopment]] was archived as [[etherpadbackup:MobileWebDevelopment]] by [[User:BryanDavis|BryanDavis]] * 23:28 [[etherpad:p/MobileWeb-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Retrospective]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/MobileWeb-Q4-Planning]] was archived as [[etherpadbackup:MobileWeb-Q4-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:27 [[etherpad:p/MobileWeb-Q3-Planning]] was archived as [[etherpadbackup:MobileWeb-Q3-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/MobileWeb-Q2-Retrospective]] was archived as [[etherpadbackup:MobileWeb-Q2-Retrospective]] by [[User:BryanDavis|BryanDavis]] * 23:26 [[etherpad:p/MobileWeb-IterationName]] was archived as [[etherpadbackup:MobileWeb-IterationName]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/MobileWeb-CodeReview-Discussion]] was archived as [[etherpadbackup:MobileWeb-CodeReview-Discussion]] by [[User:BryanDavis|BryanDavis]] * 23:25 [[etherpad:p/MobileWatchlist]] was archived as [[etherpadbackup:MobileWatchlist]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/MobileUX]] was archived as [[etherpadbackup:MobileUX]] by [[User:BryanDavis|BryanDavis]] * 23:24 [[etherpad:p/MobileTutorial]] was archived as [[etherpadbackup:MobileTutorial]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/MobileToCore]] was archived as [[etherpadbackup:MobileToCore]] by [[User:BryanDavis|BryanDavis]] * 23:23 [[etherpad:p/MobileTechDays]] was archived as [[etherpadbackup:MobileTechDays]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/MobileTeamLunchHangout]] was archived as [[etherpadbackup:MobileTeamLunchHangout]] by [[User:BryanDavis|BryanDavis]] * 23:22 [[etherpad:p/MobileTeamHangout]] was archived as [[etherpadbackup:MobileTeamHangout]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/MobileShowcase]] was archived as [[etherpadbackup:MobileShowcase]] by [[User:BryanDavis|BryanDavis]] * 23:21 [[etherpad:p/MobileResourcePlanning]] was archived as [[etherpadbackup:MobileResourcePlanning]] by [[User:BryanDavis|BryanDavis]] * 23:20 [[etherpad:p/MobileQ1_2014_2015]] was archived as [[etherpadbackup:MobileQ1_2014_2015]] by [[User:BryanDavis|BryanDavis]] * 23:20 [[etherpad:p/MobileProjectPlanning-Sept2012]] was archived as [[etherpadbackup:MobileProjectPlanning-Sept2012]] by [[User:BryanDavis|BryanDavis]] * 23:19 [[etherpad:p/MobilePageLoadStats]] was archived as [[etherpadbackup:MobilePageLoadStats]] by [[User:BryanDavis|BryanDavis]] * 23:19 [[etherpad:p/MobileOps]] was archived as [[etherpadbackup:MobileOps]] by [[User:BryanDavis|BryanDavis]] * 23:18 [[etherpad:p/MobileOperatingSystems]] was archived as [[etherpadbackup:MobileOperatingSystems]] by [[User:BryanDavis|BryanDavis]] * 23:18 [[etherpad:p/MobileFrontendCore]] was archived as [[etherpadbackup:MobileFrontendCore]] by [[User:BryanDavis|BryanDavis]] * 23:17 [[etherpad:p/MobileFrontendChanges]] was archived as [[etherpadbackup:MobileFrontendChanges]] by [[User:BryanDavis|BryanDavis]] * 23:17 [[etherpad:p/MobileFrontend-core]] was archived as [[etherpadbackup:MobileFrontend-core]] by [[User:BryanDavis|BryanDavis]] * 23:16 [[etherpad:p/MobileFrontend-Deployment-Issues-20120423]] was archived as [[etherpadbackup:MobileFrontend-Deployment-Issues-20120423]] by [[User:BryanDavis|BryanDavis]] * 23:16 [[etherpad:p/MobileFrontend-20120507-deployment-issues]] was archived as [[etherpadbackup:MobileFrontend-20120507-deployment-issues]] by [[User:BryanDavis|BryanDavis]] * 23:15 [[etherpad:p/MobileDeploymentPainPoints]] was archived as [[etherpadbackup:MobileDeploymentPainPoints]] by [[User:BryanDavis|BryanDavis]] * 23:15 [[etherpad:p/MobileChangeLog]] was archived as [[etherpadbackup:MobileChangeLog]] by [[User:BryanDavis|BryanDavis]] * 23:14 [[etherpad:p/MobileAsks]] was archived as [[etherpadbackup:MobileAsks]] by [[User:BryanDavis|BryanDavis]] * 23:13 [[etherpad:p/MobileArchitectureReview]] was archived as [[etherpadbackup:MobileArchitectureReview]] by [[User:BryanDavis|BryanDavis]] * 23:13 [[etherpad:p/MobileAppTeam]] was archived as [[etherpadbackup:MobileAppTeam]] by [[User:BryanDavis|BryanDavis]] * 23:12 [[etherpad:p/MobileAppRetrospective]] was archived as [[etherpadbackup:MobileAppRetrospective]] by [[User:BryanDavis|BryanDavis]] * 23:12 [[etherpad:p/MobileAppReleaseManger]] was archived as [[etherpadbackup:MobileAppReleaseManger]] by [[User:BryanDavis|BryanDavis]] * 23:11 [[etherpad:p/Mobile23Oct12]] was archived as [[etherpadbackup:Mobile23Oct12]] by [[User:BryanDavis|BryanDavis]] * 23:11 [[etherpad:p/Mobile-WMDS]] was archived as [[etherpadbackup:Mobile-WMDS]] by [[User:BryanDavis|BryanDavis]] * 23:10 [[etherpad:p/Mobile-VE-Integration-Planning]] was archived as [[etherpadbackup:Mobile-VE-Integration-Planning]] by [[User:BryanDavis|BryanDavis]] * 23:10 [[etherpad:p/Mobile-RL]] was archived as [[etherpadbackup:Mobile-RL]] by [[User:BryanDavis|BryanDavis]] * 23:09 [[etherpad:p/Mobile-20navigation-20planning]] was archived as [[etherpadbackup:Mobile-20navigation-20planning]] by [[User:BryanDavis|BryanDavis]] * 23:09 [[etherpad:p/Mm3uNugLtk]] was archived as [[etherpadbackup:Mm3uNugLtk]] by [[User:BryanDavis|BryanDavis]] * 23:08 [[etherpad:p/Metrics_page]] was archived as [[etherpadbackup:Metrics_page]] by [[User:BryanDavis|BryanDavis]] * 23:08 [[etherpad:p/Meetup]] was archived as [[etherpadbackup:Meetup]] by [[User:BryanDavis|BryanDavis]] * 23:07 [[etherpad:p/MediaWiki_Workshop_Buea_-_Participants_emails]] was archived as [[etherpadbackup:MediaWiki_Workshop_Buea_-_Participants_emails]] by [[User:BryanDavis|BryanDavis]] * 23:07 [[etherpad:p/MediaWiki_UI_Standardization_hack4]] was archived as [[etherpadbackup:MediaWiki_UI_Standardization_hack4]] by [[User:BryanDavis|BryanDavis]] * 23:06 [[etherpad:p/McUX]] was archived as [[etherpadbackup:McUX]] by [[User:BryanDavis|BryanDavis]] * 23:06 [[etherpad:p/Maps_pad]] was archived as [[etherpadbackup:Maps_pad]] by [[User:BryanDavis|BryanDavis]] * 23:05 [[etherpad:p/MWDS2015_multi-device_world]] was archived as [[etherpadbackup:MWDS2015_multi-device_world]] by [[User:BryanDavis|BryanDavis]] * 23:05 [[etherpad:p/MWDS2015_developer_documentation]] was archived as [[etherpadbackup:MWDS2015_developer_documentation]] by [[User:BryanDavis|BryanDavis]] * 23:04 [[etherpad:p/MWCore2015Jan]] was archived as [[etherpadbackup:MWCore2015Jan]] by [[User:BryanDavis|BryanDavis]] * 23:04 [[etherpad:p/MFCommitGuidelines]] was archived as [[etherpadbackup:MFCommitGuidelines]] by [[User:BryanDavis|BryanDavis]] * 23:03 [[etherpad:p/MF-betalabs-planning]] was archived as [[etherpadbackup:MF-betalabs-planning]] by [[User:BryanDavis|BryanDavis]] * 23:03 [[etherpad:p/MDS_2015_SOA_plenary]] was archived as [[etherpadbackup:MDS_2015_SOA_plenary]] by [[User:BryanDavis|BryanDavis]] * 23:02 [[etherpad:p/MDS_2015_SOA_and_operations]] was archived as [[etherpadbackup:MDS_2015_SOA_and_operations]] by [[User:BryanDavis|BryanDavis]] * 23:02 [[etherpad:p/Luke081515]] was archived as [[etherpadbackup:Luke081515]] by [[User:BryanDavis|BryanDavis]] * 23:01 [[etherpad:p/Logout_Confirmation_DRC]] was archived as [[etherpadbackup:Logout_Confirmation_DRC]] by [[User:BryanDavis|BryanDavis]] * 23:01 [[etherpad:p/LibrarizationRFC]] was archived as [[etherpadbackup:LibrarizationRFC]] by [[User:BryanDavis|BryanDavis]] * 23:00 [[etherpad:p/Libera-channel-queue]] was archived as [[etherpadbackup:Libera-channel-queue]] by [[User:BryanDavis|BryanDavis]] * 23:00 [[etherpad:p/Legends_of_Lavanya]] was archived as [[etherpadbackup:Legends_of_Lavanya]] by [[User:BryanDavis|BryanDavis]] * 22:59 [[etherpad:p/LabsUpgradePlans]] was archived as [[etherpadbackup:LabsUpgradePlans]] by [[User:BryanDavis|BryanDavis]] * 22:59 [[etherpad:p/LabsIrcConf]] was archived as [[etherpadbackup:LabsIrcConf]] by [[User:BryanDavis|BryanDavis]] * 22:58 [[etherpad:p/LabsAPI]] was archived as [[etherpadbackup:LabsAPI]] by [[User:BryanDavis|BryanDavis]] * 22:58 [[etherpad:p/Labs-Incident-Report-20015-03-31]] was archived as [[etherpadbackup:Labs-Incident-Report-20015-03-31]] by [[User:BryanDavis|BryanDavis]] * 22:57 [[etherpad:p/Labs-2017-03-14]] was archived as [[etherpadbackup:Labs-2017-03-14]] by [[User:BryanDavis|BryanDavis]] * 22:57 [[etherpad:p/Labs-2017-03-06]] was archived as [[etherpadbackup:Labs-2017-03-06]] by [[User:BryanDavis|BryanDavis]] * 22:56 [[etherpad:p/Labs-2017-02-22]] was archived as [[etherpadbackup:Labs-2017-02-22]] by [[User:BryanDavis|BryanDavis]] * 22:56 [[etherpad:p/Labs-2017-01-23]] was archived as [[etherpadbackup:Labs-2017-01-23]] by [[User:BryanDavis|BryanDavis]] * 22:55 [[etherpad:p/Labs-2016-12-12]] was archived as [[etherpadbackup:Labs-2016-12-12]] by [[User:BryanDavis|BryanDavis]] * 22:54 [[etherpad:p/Labs-2016-11-07]] was archived as [[etherpadbackup:Labs-2016-11-07]] by [[User:BryanDavis|BryanDavis]] * 22:54 [[etherpad:p/Labs-2016-10-31]] was archived as [[etherpadbackup:Labs-2016-10-31]] by [[User:BryanDavis|BryanDavis]] * 22:53 [[etherpad:p/Labs-2016-09-19]] was archived as [[etherpadbackup:Labs-2016-09-19]] by [[User:BryanDavis|BryanDavis]] * 22:53 [[etherpad:p/Labs-2016-09-06]] was archived as [[etherpadbackup:Labs-2016-09-06]] by [[User:BryanDavis|BryanDavis]] * 22:52 [[etherpad:p/Labs-2016-08-29]] was archived as [[etherpadbackup:Labs-2016-08-29]] by [[User:BryanDavis|BryanDavis]] * 22:52 [[etherpad:p/Labs-2016-08-01]] was archived as [[etherpadbackup:Labs-2016-08-01]] by [[User:BryanDavis|BryanDavis]] * 22:51 [[etherpad:p/Labs-2016-07-25]] was archived as [[etherpadbackup:Labs-2016-07-25]] by [[User:BryanDavis|BryanDavis]] * 22:51 [[etherpad:p/Labs-2016-07-11]] was archived as [[etherpadbackup:Labs-2016-07-11]] by [[User:BryanDavis|BryanDavis]] * 22:50 [[etherpad:p/Labs-2016-06-06]] was archived as [[etherpadbackup:Labs-2016-06-06]] by [[User:BryanDavis|BryanDavis]] * 22:50 [[etherpad:p/Labs-2016-05-23]] was archived as [[etherpadbackup:Labs-2016-05-23]] by [[User:BryanDavis|BryanDavis]] * 22:49 [[etherpad:p/Labs-2016-05-16]] was archived as [[etherpadbackup:Labs-2016-05-16]] by [[User:BryanDavis|BryanDavis]] * 22:49 [[etherpad:p/Labs-2016-05-09]] was archived as [[etherpadbackup:Labs-2016-05-09]] by [[User:BryanDavis|BryanDavis]] * 22:48 [[etherpad:p/Labs-2016-05-02]] was archived as [[etherpadbackup:Labs-2016-05-02]] by [[User:BryanDavis|BryanDavis]] * 22:48 [[etherpad:p/Labs-2016-04-25]] was archived as [[etherpadbackup:Labs-2016-04-25]] by [[User:BryanDavis|BryanDavis]] * 22:47 [[etherpad:p/Labs-20150602]] was archived as [[etherpadbackup:Labs-20150602]] by [[User:BryanDavis|BryanDavis]] * 22:47 [[etherpad:p/L]] was archived as [[etherpadbackup:L]] by [[User:BryanDavis|BryanDavis]] * 22:46 [[etherpad:p/Kay@31]] was archived as [[etherpadbackup:Kay@31]] by [[User:BryanDavis|BryanDavis]] * 22:46 [[etherpad:p/Kafka-udp2log]] was archived as [[etherpadbackup:Kafka-udp2log]] by [[User:BryanDavis|BryanDavis]] * 22:45 [[etherpad:p/K]] was archived as [[etherpadbackup:K]] by [[User:BryanDavis|BryanDavis]] * 22:45 [[etherpad:p/J]] was archived as [[etherpadbackup:J]] by [[User:BryanDavis|BryanDavis]] * 22:44 [[etherpad:p/Introduction_to_Phabricator_(WikiCon)]] was archived as [[etherpadbackup:Introduction_to_Phabricator_(WikiCon)]] by [[User:BryanDavis|BryanDavis]] * 22:44 [[etherpad:p/Intro_a_las_tec_del_movimiento_Wikimedia]] was archived as [[etherpadbackup:Intro_a_las_tec_del_movimiento_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 22:43 [[etherpad:p/InlineScripting]] was archived as [[etherpadbackup:InlineScripting]] by [[User:BryanDavis|BryanDavis]] * 22:43 [[etherpad:p/Incident-Replay-Sessions]] was archived as [[etherpadbackup:Incident-Replay-Sessions]] by [[User:BryanDavis|BryanDavis]] * 22:42 [[etherpad:p/IlszcZZEEq]] was archived as [[etherpadbackup:IlszcZZEEq]] by [[User:BryanDavis|BryanDavis]] * 22:42 [[etherpad:p/Icon_standardisation]] was archived as [[etherpadbackup:Icon_standardisation]] by [[User:BryanDavis|BryanDavis]] * 22:41 [[etherpad:p/INGLW4jCQk]] was archived as [[etherpadbackup:INGLW4jCQk]] by [[User:BryanDavis|BryanDavis]] * 22:41 [[etherpad:p/IBrokeWikipediaList]] was archived as [[etherpadbackup:IBrokeWikipediaList]] by [[User:BryanDavis|BryanDavis]] * 22:40 [[etherpad:p/I]] was archived as [[etherpadbackup:I]] by [[User:BryanDavis|BryanDavis]] * 22:39 [[etherpad:p/HuggleCon-2-planning]] was archived as [[etherpadbackup:HuggleCon-2-planning]] by [[User:BryanDavis|BryanDavis]] * 22:39 [[etherpad:p/HuggleCon-1-participants-list]] was archived as [[etherpadbackup:HuggleCon-1-participants-list]] by [[User:BryanDavis|BryanDavis]] * 22:38 [[etherpad:p/Huggle-dev-pad]] was archived as [[etherpadbackup:Huggle-dev-pad]] by [[User:BryanDavis|BryanDavis]] * 22:38 [[etherpad:p/Huggle-IRC-conference]] was archived as [[etherpadbackup:Huggle-IRC-conference]] by [[User:BryanDavis|BryanDavis]] * 22:37 [[etherpad:p/Huggle]] was archived as [[etherpadbackup:Huggle]] by [[User:BryanDavis|BryanDavis]] * 22:37 [[etherpad:p/HowToReleaseAnApp]] was archived as [[etherpadbackup:HowToReleaseAnApp]] by [[User:BryanDavis|BryanDavis]] * 22:36 [[etherpad:p/Hiera]] was archived as [[etherpadbackup:Hiera]] by [[User:BryanDavis|BryanDavis]] * 22:36 [[etherpad:p/HadoopEtherpad]] was archived as [[etherpadbackup:HadoopEtherpad]] by [[User:BryanDavis|BryanDavis]] * 22:35 [[etherpad:p/Hackathon_Showcase_&_Coolest_Tool_Awards]] was archived as [[etherpadbackup:Hackathon_Showcase_&_Coolest_Tool_Awards]] by [[User:BryanDavis|BryanDavis]] * 22:35 [[etherpad:p/HKzL6x5e6z]] was archived as [[etherpadbackup:HKzL6x5e6z]] by [[User:BryanDavis|BryanDavis]] * 22:34 [[etherpad:p/HJRCD1b81bHXm-ZS-l9w]] was archived as [[etherpadbackup:HJRCD1b81bHXm-ZS-l9w]] by [[User:BryanDavis|BryanDavis]] * 22:34 [[etherpad:p/Gwods9UzuoEMR79NM8nl]] was archived as [[etherpadbackup:Gwods9UzuoEMR79NM8nl]] by [[User:BryanDavis|BryanDavis]] * 22:33 [[etherpad:p/GroupSelectOption]] was archived as [[etherpadbackup:GroupSelectOption]] by [[User:BryanDavis|BryanDavis]] * 22:33 [[etherpad:p/Google_Slack_Zoom_YouTube_bugs_in_Movement]] was archived as [[etherpadbackup:Google_Slack_Zoom_YouTube_bugs_in_Movement]] by [[User:BryanDavis|BryanDavis]] * 22:32 [[etherpad:p/GitWorkflow]] was archived as [[etherpadbackup:GitWorkflow]] by [[User:BryanDavis|BryanDavis]] * 22:32 [[etherpad:p/Git-migration-tools]] was archived as [[etherpadbackup:Git-migration-tools]] by [[User:BryanDavis|BryanDavis]] * 22:31 [[etherpad:p/Git-migration]] was archived as [[etherpadbackup:Git-migration]] by [[User:BryanDavis|BryanDavis]] * 22:31 [[etherpad:p/Git-Feb13]] was archived as [[etherpadbackup:Git-Feb13]] by [[User:BryanDavis|BryanDavis]] * 22:30 [[etherpad:p/Gerrit_login_problems]] was archived as [[etherpadbackup:Gerrit_login_problems]] by [[User:BryanDavis|BryanDavis]] * 22:30 [[etherpad:p/Gerrit_API_Wikimedia]] was archived as [[etherpadbackup:Gerrit_API_Wikimedia]] by [[User:BryanDavis|BryanDavis]] * 22:29 [[etherpad:p/GerritCleanupOrg]] was archived as [[etherpadbackup:GerritCleanupOrg]] by [[User:BryanDavis|BryanDavis]] * 22:29 [[etherpad:p/GerritAccounts]] was archived as [[etherpadbackup:GerritAccounts]] by [[User:BryanDavis|BryanDavis]] * 22:28 [[etherpad:p/GeoData]] was archived as [[etherpadbackup:GeoData]] by [[User:BryanDavis|BryanDavis]] * 22:28 [[etherpad:p/Gather_Q4]] was archived as [[etherpadbackup:Gather_Q4]] by [[User:BryanDavis|BryanDavis]] * 22:27 [[etherpad:p/G__________________]] was archived as [[etherpadbackup:G__________________]] by [[User:BryanDavis|BryanDavis]] * 22:27 [[etherpad:p/GWToolsetReview]] was archived as [[etherpadbackup:GWToolsetReview]] by [[User:BryanDavis|BryanDavis]] * 22:26 [[etherpad:p/G]] was archived as [[etherpadbackup:G]] by [[User:BryanDavis|BryanDavis]] * 22:26 [[etherpad:p/FutureOfEditingTalk]] was archived as [[etherpadbackup:FutureOfEditingTalk]] by [[User:BryanDavis|BryanDavis]] * 22:25 [[etherpad:p/FrontEndDevOnboarding-bahodir]] was archived as [[etherpadbackup:FrontEndDevOnboarding-bahodir]] by [[User:BryanDavis|BryanDavis]] * 22:25 [[etherpad:p/FrontEndDevOnboarding]] was archived as [[etherpadbackup:FrontEndDevOnboarding]] by [[User:BryanDavis|BryanDavis]] * 22:24 [[etherpad:p/France_-_communes_-_Data_.tab]] was archived as [[etherpadbackup:France_-_communes_-_Data_.tab]] by [[User:BryanDavis|BryanDavis]] * 22:24 [[etherpad:p/Foo]] was archived as [[etherpadbackup:Foo]] by [[User:BryanDavis|BryanDavis]] * 22:23 [[etherpad:p/FlowTodo]] was archived as [[etherpadbackup:FlowTodo]] by [[User:BryanDavis|BryanDavis]] * 22:23 [[etherpad:p/FlowAsAService]] was archived as [[etherpadbackup:FlowAsAService]] by [[User:BryanDavis|BryanDavis]] * 22:22 [[etherpad:p/Flow-ops-l_mail]] was archived as [[etherpadbackup:Flow-ops-l_mail]] by [[User:BryanDavis|BryanDavis]] * 22:21 [[etherpad:p/First_URLs]] was archived as [[etherpadbackup:First_URLs]] by [[User:BryanDavis|BryanDavis]] * 22:21 [[etherpad:p/Feb6Outage]] was archived as [[etherpadbackup:Feb6Outage]] by [[User:BryanDavis|BryanDavis]] * 22:20 [[etherpad:p/Feb-6-Issues]] was archived as [[etherpadbackup:Feb-6-Issues]] by [[User:BryanDavis|BryanDavis]] * 22:20 [[etherpad:p/FeaturesTeam2012-W15]] was archived as [[etherpadbackup:FeaturesTeam2012-W15]] by [[User:BryanDavis|BryanDavis]] * 22:19 [[etherpad:p/FeaturesTeam2012-W14]] was archived as [[etherpadbackup:FeaturesTeam2012-W14]] by [[User:BryanDavis|BryanDavis]] * 22:19 [[etherpad:p/FeaturesTeam2012-W13]] was archived as [[etherpadbackup:FeaturesTeam2012-W13]] by [[User:BryanDavis|BryanDavis]] * 22:18 [[etherpad:p/FeaturesTeam2012-W08]] was archived as [[etherpadbackup:FeaturesTeam2012-W08]] by [[User:BryanDavis|BryanDavis]] * 22:18 [[etherpad:p/FeaturesTeam2012-W07]] was archived as [[etherpadbackup:FeaturesTeam2012-W07]] by [[User:BryanDavis|BryanDavis]] * 22:17 [[etherpad:p/FeaturesTeam2012-W06]] was archived as [[etherpadbackup:FeaturesTeam2012-W06]] by [[User:BryanDavis|BryanDavis]] * 22:17 [[etherpad:p/FeaturesTeam2012-W03]] was archived as [[etherpadbackup:FeaturesTeam2012-W03]] by [[User:BryanDavis|BryanDavis]] * 22:16 [[etherpad:p/FeaturesTeam2012-W011]] was archived as [[etherpadbackup:FeaturesTeam2012-W011]] by [[User:BryanDavis|BryanDavis]] * 22:16 [[etherpad:p/FeaturesTeam20111227]] was archived as [[etherpadbackup:FeaturesTeam20111227]] by [[User:BryanDavis|BryanDavis]] * 22:15 [[etherpad:p/FeaturesTeam20111213]] was archived as [[etherpadbackup:FeaturesTeam20111213]] by [[User:BryanDavis|BryanDavis]] * 22:15 [[etherpad:p/FeaturesTeam20111206]] was archived as [[etherpadbackup:FeaturesTeam20111206]] by [[User:BryanDavis|BryanDavis]] * 22:14 [[etherpad:p/FeaturesTeam-Preload]] was archived as [[etherpadbackup:FeaturesTeam-Preload]] by [[User:BryanDavis|BryanDavis]] * 22:14 [[etherpad:p/Farm101]] was archived as [[etherpadbackup:Farm101]] by [[User:BryanDavis|BryanDavis]] * 22:13 [[etherpad:p/FRAnalyticsQ2FY16]] was archived as [[etherpadbackup:FRAnalyticsQ2FY16]] by [[User:BryanDavis|BryanDavis]] * 22:13 [[etherpad:p/FG3gMpB1qc]] was archived as [[etherpadbackup:FG3gMpB1qc]] by [[User:BryanDavis|BryanDavis]] * 22:12 [[etherpad:p/F2pDHw25sm7eJS1Q3Tk2]] was archived as [[etherpadbackup:F2pDHw25sm7eJS1Q3Tk2]] by [[User:BryanDavis|BryanDavis]] * 22:12 [[etherpad:p/F2gsk6WTo1]] was archived as [[etherpadbackup:F2gsk6WTo1]] by [[User:BryanDavis|BryanDavis]] * 22:11 [[etherpad:p/F]] was archived as [[etherpadbackup:F]] by [[User:BryanDavis|BryanDavis]] * 22:11 [[etherpad:p/ErAnycbwZA]] was archived as [[etherpadbackup:ErAnycbwZA]] by [[User:BryanDavis|BryanDavis]] * 22:10 [[etherpad:p/EpicIndiaItinerary]] was archived as [[etherpadbackup:EpicIndiaItinerary]] by [[User:BryanDavis|BryanDavis]] * 22:10 [[etherpad:p/Encuentro_técnico_hispanohablante_Wikimania_2022]] was archived as [[etherpadbackup:Encuentro_técnico_hispanohablante_Wikimania_2022]] by [[User:BryanDavis|BryanDavis]] * 22:09 [[etherpad:p/EchoNotification]] was archived as [[etherpadbackup:EchoNotification]] by [[User:BryanDavis|BryanDavis]] * 22:09 [[etherpad:p/EQIAD-rollout-sequence]] was archived as [[etherpadbackup:EQIAD-rollout-sequence]] by [[User:BryanDavis|BryanDavis]] * 22:08 [[etherpad:p/EL_python]] was archived as [[etherpadbackup:EL_python]] by [[User:BryanDavis|BryanDavis]] * 22:08 [[etherpad:p/E3Analytics]] was archived as [[etherpadbackup:E3Analytics]] by [[User:BryanDavis|BryanDavis]] * 22:07 [[etherpad:p/E3-2013-03-07-deploy]] was archived as [[etherpadbackup:E3-2013-03-07-deploy]] by [[User:BryanDavis|BryanDavis]] * 22:07 [[etherpad:p/DrinkingShoppingList]] was archived as [[etherpadbackup:DrinkingShoppingList]] by [[User:BryanDavis|BryanDavis]] * 22:06 [[etherpad:p/DownWithTwentyPercent]] was archived as [[etherpadbackup:DownWithTwentyPercent]] by [[User:BryanDavis|BryanDavis]] * 22:06 [[etherpad:p/Discussion__Redefine_Wikimedia_Hackathons]] was archived as [[etherpadbackup:Discussion__Redefine_Wikimedia_Hackathons]] by [[User:BryanDavis|BryanDavis]] * 22:05 [[etherpad:p/DevelopingMobileApps]] was archived as [[etherpadbackup:DevelopingMobileApps]] by [[User:BryanDavis|BryanDavis]] * 22:05 [[etherpad:p/DevSummitMain]] was archived as [[etherpadbackup:DevSummitMain]] by [[User:BryanDavis|BryanDavis]] * 22:04 [[etherpad:p/DevSum2015_designresearch]] was archived as [[etherpadbackup:DevSum2015_designresearch]] by [[User:BryanDavis|BryanDavis]] * 22:03 [[etherpad:p/DeploymentPrepReConfig]] was archived as [[etherpadbackup:DeploymentPrepReConfig]] by [[User:BryanDavis|BryanDavis]] * 22:03 [[etherpad:p/DSEorCDH4]] was archived as [[etherpadbackup:DSEorCDH4]] by [[User:BryanDavis|BryanDavis]] * 22:02 [[etherpad:p/DRworkshopParkingLot]] was archived as [[etherpadbackup:DRworkshopParkingLot]] by [[User:BryanDavis|BryanDavis]] * 22:02 [[etherpad:p/DRMethodsMenuSite]] was archived as [[etherpadbackup:DRMethodsMenuSite]] by [[User:BryanDavis|BryanDavis]] * 22:01 [[etherpad:p/ContinuousIntegration]] was archived as [[etherpadbackup:ContinuousIntegration]] by [[User:BryanDavis|BryanDavis]] * 22:01 [[etherpad:p/CollectionsNameA]] was archived as [[etherpadbackup:CollectionsNameA]] by [[User:BryanDavis|BryanDavis]] * 22:00 [[etherpad:p/Code-in_student_meeting]] was archived as [[etherpadbackup:Code-in_student_meeting]] by [[User:BryanDavis|BryanDavis]] * 22:00 [[etherpad:p/CmMGbzxbZw]] was archived as [[etherpadbackup:CmMGbzxbZw]] by [[User:BryanDavis|BryanDavis]] * 21:59 [[etherpad:p/Chromium-Upstream]] was archived as [[etherpadbackup:Chromium-Upstream]] by [[User:BryanDavis|BryanDavis]] * 21:59 [[etherpad:p/CentralNoticeDiscussion]] was archived as [[etherpadbackup:CentralNoticeDiscussion]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/CentralNotice]] was archived as [[etherpadbackup:CentralNotice]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/CampaignAssociatedMixins]] was archived as [[etherpadbackup:CampaignAssociatedMixins]] by [[User:BryanDavis|BryanDavis]] * 21:57 [[etherpad:p/C_________]] was archived as [[etherpadbackup:C_________]] by [[User:BryanDavis|BryanDavis]] * 21:57 [[etherpad:p/CZg-suME0Sx-nnCfKIEW]] was archived as [[etherpadbackup:CZg-suME0Sx-nnCfKIEW]] by [[User:BryanDavis|BryanDavis]] * 21:56 [[etherpad:p/CSF_Katie]] was archived as [[etherpadbackup:CSF_Katie]] by [[User:BryanDavis|BryanDavis]] * 21:56 [[etherpad:p/CEE-l10n-levels]] was archived as [[etherpadbackup:CEE-l10n-levels]] by [[User:BryanDavis|BryanDavis]] * 21:55 [[etherpad:p/C-Lessons]] was archived as [[etherpadbackup:C-Lessons]] by [[User:BryanDavis|BryanDavis]] * 21:55 [[etherpad:p/BotLicensing]] was archived as [[etherpadbackup:BotLicensing]] by [[User:BryanDavis|BryanDavis]] * 21:54 [[etherpad:p/BetaClusterpriorityworksync]] was archived as [[etherpadbackup:BetaClusterpriorityworksync]] by [[User:BryanDavis|BryanDavis]] * 21:54 [[etherpad:p/ArchitectureCommittee-2015-02-04]] was archived as [[etherpadbackup:ArchitectureCommittee-2015-02-04]] by [[User:BryanDavis|BryanDavis]] * 21:53 [[etherpad:p/ArchDoc]] was archived as [[etherpadbackup:ArchDoc]] by [[User:BryanDavis|BryanDavis]] * 21:53 [[etherpad:p/AppsReadership]] was archived as [[etherpadbackup:AppsReadership]] by [[User:BryanDavis|BryanDavis]] * 21:52 [[etherpad:p/AppsQ120142015]] was archived as [[etherpadbackup:AppsQ120142015]] by [[User:BryanDavis|BryanDavis]] * 21:52 [[etherpad:p/AppsPhabelloMigration]] was archived as [[etherpadbackup:AppsPhabelloMigration]] by [[User:BryanDavis|BryanDavis]] * 21:51 [[etherpad:p/AppsNav]] was archived as [[etherpadbackup:AppsNav]] by [[User:BryanDavis|BryanDavis]] * 21:51 [[etherpad:p/AppsContentService]] was archived as [[etherpadbackup:AppsContentService]] by [[User:BryanDavis|BryanDavis]] * 21:50 [[etherpad:p/AppUserTesting]] was archived as [[etherpadbackup:AppUserTesting]] by [[User:BryanDavis|BryanDavis]] * 21:50 [[etherpad:p/App-20design-20tweaks]] was archived as [[etherpadbackup:App-20design-20tweaks]] by [[User:BryanDavis|BryanDavis]] * 21:49 [[etherpad:p/Android_front-end_dev_task]] was archived as [[etherpadbackup:Android_front-end_dev_task]] by [[User:BryanDavis|BryanDavis]] * 21:48 [[etherpad:p/Analytics_all_staff]] was archived as [[etherpadbackup:Analytics_all_staff]] by [[User:BryanDavis|BryanDavis]] * 21:48 [[etherpad:p/Analytics_Wikimania]] was archived as [[etherpadbackup:Analytics_Wikimania]] by [[User:BryanDavis|BryanDavis]] * 21:47 [[etherpad:p/AnalyticsPlanningMeetingNotes]] was archived as [[etherpadbackup:AnalyticsPlanningMeetingNotes]] by [[User:BryanDavis|BryanDavis]] * 21:47 [[etherpad:p/AnalyticsMilestonesAndPriorities]] was archived as [[etherpadbackup:AnalyticsMilestonesAndPriorities]] by [[User:BryanDavis|BryanDavis]] * 21:46 [[etherpad:p/AnalyticsDevStaff]] was archived as [[etherpadbackup:AnalyticsDevStaff]] by [[User:BryanDavis|BryanDavis]] * 21:46 [[etherpad:p/Analytics-Reseach]] was archived as [[etherpadbackup:Analytics-Reseach]] by [[User:BryanDavis|BryanDavis]] * 21:45 [[etherpad:p/Analytics-Pixel-Service]] was archived as [[etherpadbackup:Analytics-Pixel-Service]] by [[User:BryanDavis|BryanDavis]] * 21:45 [[etherpad:p/Alpha-beta_cleaning]] was archived as [[etherpadbackup:Alpha-beta_cleaning]] by [[User:BryanDavis|BryanDavis]] * 21:44 [[etherpad:p/Adv_Browser_Tests_-_T86593]] was archived as [[etherpadbackup:Adv_Browser_Tests_-_T86593]] by [[User:BryanDavis|BryanDavis]] * 21:44 [[etherpad:p/Adedolapo_-_survey_for_content_Translation_users]] was archived as [[etherpadbackup:Adedolapo_-_survey_for_content_Translation_users]] by [[User:BryanDavis|BryanDavis]] * 21:43 [[etherpad:p/Accessibility_WMDS]] was archived as [[etherpadbackup:Accessibility_WMDS]] by [[User:BryanDavis|BryanDavis]] * 21:43 [[etherpad:p/AWMD_IRC-Meeting-13]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-13]] by [[User:BryanDavis|BryanDavis]] * 21:42 [[etherpad:p/AWMD_IRC-Meeting-12]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-12]] by [[User:BryanDavis|BryanDavis]] * 21:42 [[etherpad:p/AWMD_IRC-Meeting-11]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-11]] by [[User:BryanDavis|BryanDavis]] * 21:41 [[etherpad:p/AWMD_IRC-Meeting-10]] was archived as [[etherpadbackup:AWMD_IRC-Meeting-10]] by [[User:BryanDavis|BryanDavis]] * 21:41 [[etherpad:p/API-blog-post]] was archived as [[etherpadbackup:API-blog-post]] by [[User:BryanDavis|BryanDavis]] * 21:40 [[etherpad:p/AFT5LaunchTests]] was archived as [[etherpadbackup:AFT5LaunchTests]] by [[User:BryanDavis|BryanDavis]] * 21:40 [[etherpad:p/AFT5]] was archived as [[etherpadbackup:AFT5]] by [[User:BryanDavis|BryanDavis]] * 21:39 [[etherpad:p/9RFl8QljDZ]] was archived as [[etherpadbackup:9RFl8QljDZ]] by [[User:BryanDavis|BryanDavis]] * 21:39 [[etherpad:p/92]] was archived as [[etherpadbackup:92]] by [[User:BryanDavis|BryanDavis]] * 21:38 [[etherpad:p/8r8ihQ7GpX]] was archived as [[etherpadbackup:8r8ihQ7GpX]] by [[User:BryanDavis|BryanDavis]] * 21:38 [[etherpad:p/8D1ml5Q1Sb]] was archived as [[etherpadbackup:8D1ml5Q1Sb]] by [[User:BryanDavis|BryanDavis]] * 21:37 [[etherpad:p/88952]] was archived as [[etherpadbackup:88952]] by [[User:BryanDavis|BryanDavis]] * 21:37 [[etherpad:p/7hDQxehZZW]] was archived as [[etherpadbackup:7hDQxehZZW]] by [[User:BryanDavis|BryanDavis]] * 21:36 [[etherpad:p/7YAmoRjGSAvfJrj9V9sD]] was archived as [[etherpadbackup:7YAmoRjGSAvfJrj9V9sD]] by [[User:BryanDavis|BryanDavis]] * 21:36 [[etherpad:p/7FtE43Tuhu]] was archived as [[etherpadbackup:7FtE43Tuhu]] by [[User:BryanDavis|BryanDavis]] * 21:35 [[etherpad:p/77SMYoPXek]] was archived as [[etherpadbackup:77SMYoPXek]] by [[User:BryanDavis|BryanDavis]] * 21:35 [[etherpad:p/6gR8aSREkz]] was archived as [[etherpadbackup:6gR8aSREkz]] by [[User:BryanDavis|BryanDavis]] * 21:34 [[etherpad:p/58040]] was archived as [[etherpadbackup:58040]] by [[User:BryanDavis|BryanDavis]] * 21:34 [[etherpad:p/503-eqiad-cirrussearch]] was archived as [[etherpadbackup:503-eqiad-cirrussearch]] by [[User:BryanDavis|BryanDavis]] * 21:33 [[etherpad:p/4pKxno7DAI]] was archived as [[etherpadbackup:4pKxno7DAI]] by [[User:BryanDavis|BryanDavis]] * 21:33 [[etherpad:p/4J8fHuL0yB]] was archived as [[etherpadbackup:4J8fHuL0yB]] by [[User:BryanDavis|BryanDavis]] * 21:32 [[etherpad:p/4.1_launch_post-mortem]] was archived as [[etherpadbackup:4.1_launch_post-mortem]] by [[User:BryanDavis|BryanDavis]] * 21:32 [[etherpad:p/20percent]] was archived as [[etherpadbackup:20percent]] by [[User:BryanDavis|BryanDavis]] * 21:31 [[etherpad:p/2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] was archived as [[etherpadbackup:2026.03_Wikibase_SPARQL_Prefix_@_WMHack]] by [[User:BryanDavis|BryanDavis]] * 21:30 [[etherpad:p/2025-10-workers-reboot]] was archived as [[etherpadbackup:2025-10-workers-reboot]] by [[User:BryanDavis|BryanDavis]] * 21:30 [[etherpad:p/2025-04_toolforge_monthly_update]] was archived as [[etherpadbackup:2025-04_toolforge_monthly_update]] by [[User:BryanDavis|BryanDavis]] * 21:29 [[etherpad:p/2024_tekton_upgrade]] was archived as [[etherpadbackup:2024_tekton_upgrade]] by [[User:BryanDavis|BryanDavis]] * 21:29 [[etherpad:p/2021-switchdc-testing]] was archived as [[etherpadbackup:2021-switchdc-testing]] by [[User:BryanDavis|BryanDavis]] * 21:28 [[etherpad:p/2021-switchdc-prep]] was archived as [[etherpadbackup:2021-switchdc-prep]] by [[User:BryanDavis|BryanDavis]] * 21:28 [[etherpad:p/20205-06-components-api-reviews]] was archived as [[etherpadbackup:20205-06-components-api-reviews]] by [[User:BryanDavis|BryanDavis]] * 21:27 [[etherpad:p/2019-09-25-cloudvirt-load-rebalance]] was archived as [[etherpadbackup:2019-09-25-cloudvirt-load-rebalance]] by [[User:BryanDavis|BryanDavis]] * 21:27 [[etherpad:p/2019-07-23-eqiad-asw-a]] was archived as [[etherpadbackup:2019-07-23-eqiad-asw-a]] by [[User:BryanDavis|BryanDavis]] * 21:26 [[etherpad:p/2019-07-16-docker-registry]] was archived as [[etherpadbackup:2019-07-16-docker-registry]] by [[User:BryanDavis|BryanDavis]] * 21:26 [[etherpad:p/20180829-labstore-reboots]] was archived as [[etherpadbackup:20180829-labstore-reboots]] by [[User:BryanDavis|BryanDavis]] * 21:25 [[etherpad:p/20150519-Mailman]] was archived as [[etherpadbackup:20150519-Mailman]] by [[User:BryanDavis|BryanDavis]] * 21:25 [[etherpad:p/2014-antoine-june-checkin]] was archived as [[etherpadbackup:2014-antoine-june-checkin]] by [[User:BryanDavis|BryanDavis]] * 21:24 [[etherpad:p/2013-2014-Q2-MobileWeb]] was archived as [[etherpadbackup:2013-2014-Q2-MobileWeb]] by [[User:BryanDavis|BryanDavis]] * 21:24 [[etherpad:p/2012-2013-Q1-Mobile-Dept]] was archived as [[etherpadbackup:2012-2013-Q1-Mobile-Dept]] by [[User:BryanDavis|BryanDavis]] * 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]] * 21:23 [[etherpad:p/19-jun-2014-parsercache-outage]] was archived as [[etherpadbackup:19-jun-2014-parsercache-outage]] by [[User:BryanDavis|BryanDavis]] * 20:22 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] === 2026-05-03 === * 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 13:56 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 13:56 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]] * 13:14 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/123]] was archived as [[etherpadbackup:123]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/120wmf2deployment]] was archived as [[etherpadbackup:120wmf2deployment]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119triage]] was archived as [[etherpadbackup:119triage]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119deployment]] was archived as [[etherpadbackup:119deployment]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119917]] was archived as [[etherpadbackup:119917]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/119-beta]] was archived as [[etherpadbackup:119-beta]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.36-release]] was archived as [[etherpadbackup:1.36-release]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.27.0-wmf.13-issues]] was archived as [[etherpadbackup:1.27.0-wmf.13-issues]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/1.25wmf15-perf-regression]] was archived as [[etherpadbackup:1.25wmf15-perf-regression]] by [[User:BryanDavis|BryanDavis]] * 13:11 [[etherpad:p/0WdKUDRA58]] was archived as [[etherpadbackup:0WdKUDRA58]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WIKISOO3]] was archived as [[etherpadbackup:WIKISOO3]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/VertalingWikiIt]] was archived as [[etherpadbackup:VertalingWikiIt]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]] * 12:02 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/WBUG_2023_09_28]] was archived as [[etherpadbackup:WBUG_2023_09_28]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/WBUG_2023_07_27]] was archived as [[etherpadbackup:WBUG_2023_07_27]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/IRCBot-Messages]] was archived as [[etherpadbackup:IRCBot-Messages]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GLAMcampAmsterdamFri]] was archived as [[etherpadbackup:GLAMcampAmsterdamFri]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GLAM-Wiki-CH_OpenRefine_PAWS]] was archived as [[etherpadbackup:GLAM-Wiki-CH_OpenRefine_PAWS]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/GD0bnahDcM]] was archived as [[etherpadbackup:GD0bnahDcM]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/DrTrigonbotfalsepositivesfilter]] was archived as [[etherpadbackup:DrTrigonbotfalsepositivesfilter]] by [[User:BryanDavis|BryanDavis]] * 12:00 [[etherpad:p/BugTriage-Collection]] was archived as [[etherpadbackup:BugTriage-Collection]] by [[User:BryanDavis|BryanDavis]] * 09:25 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 09:18 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:46 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:28 [[etherpad:p/prodlikevagrant-hackathon-planning]] was archived as [[etherpadbackup:prodlikevagrant-hackathon-planning]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-05-02 === * 13:01 [[etherpad:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] was archived as [[etherpadbackup:p/wpcon16_Wikipedianer_%26_%E2%80%9EMateriale_Textkultu]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 07:18 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 06:58 [[etherpad:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] was archived as [[etherpadbackup:p/test-*%2C%3F%2C\%2C%2F%2C%25%2C>%2C<%2C🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-05-01 === * 18:12 [[etherpad:p/10myths]] was archived as [[etherpadbackup:p/10myths]] by [[User:Krinkle|Krinkle]] * 15:30 [[etherpad:p/test-*,?,\,/,%,>,<,🤖💬]] was archived as [[etherpadbackup:p/test-*,?,\,/,%,>,<,🤖💬]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 14:50 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:Jon Harald Søby|Jon Harald Søby]] * 14:42 [[etherpad:p/PoGo_Friendship_Codes]] was archived as [[etherpadbackup:p/PoGo_Friendship_Codes]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:51 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:16 [[etherpad:p/Wikimedia_Hackathon_2026_Opening]] was archived as [[etherpadbackup:p/Wikimedia_Hackathon_2026_Opening]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 13:07 [[etherpad:p/copyvios-todo]] was archived as [[etherpadbackup:p/copyvios-todo]] by [[User:Chlod|Chlod]] * 10:05 [[etherpad:p/hhvmdeploy]] was archived as [[etherpadbackup:p/hhvmdeploy]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-04-29 === * 00:49 [[etherpad:p/better-security-patch-management]] was archived as [[etherpadbackup:p/better-security-patch-management]] by [[User:BDavis (WMF)|BDavis (WMF)]] * 00:49 [[etherpad:p/rooks-questions-to-alex]] was archived as [[etherpadbackup:p/rooks-questions-to-alex]] by [[User:BDavis (WMF)|BDavis (WMF)]] === 2026-04-27 === * 04:44 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 03:46 [[etherpad:p/wmh2023-Overview_of_technical_areas_&_projects]] was archived as [[etherpadbackup:p/wmh2023-Overview_of_technical_areas_&_projects]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/open_meeting_COVID-19_18/04]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_18/04]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/open_meeting_COVID-19_04/05]] was archived as [[etherpadbackup:p/open_meeting_COVID-19_04/05]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/aswikimeet/july-2024]] was archived as [[etherpadbackup:p/aswikimeet/july-2024]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikisource_Meetup_29/09/2020]] was archived as [[etherpadbackup:p/Wikisource_Meetup_29/09/2020]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] was archived as [[etherpadbackup:p/Wikimedians_of_Nepal/Events/Meetup_February_2020]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/Wikimedia_LGBT+/Governance/2023-10-13]] was archived as [[etherpadbackup:p/Wikimedia_LGBT+/Governance/2023-10-13]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopD]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopD]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopC]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopC]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopB]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopB]] by [[User:BryanDavis|BryanDavis]] * 03:45 [[etherpad:p/WikiCendekia2026/WarkopA]] was archived as [[etherpadbackup:p/WikiCendekia2026/WarkopA]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Stolpersteine_nach_WikiData_(01/2025]] was archived as [[etherpadbackup:p/Stolpersteine_nach_WikiData_(01/2025]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Sommer_24/8,_1._August]] was archived as [[etherpadbackup:p/Sommer_24/8,_1._August]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] was archived as [[etherpadbackup:p/SVG_Translation_Campaign_2019_India/Event/Patiala]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Meetup/Cape_Town/Cape_Town_22]] was archived as [[etherpadbackup:p/Meetup/Cape_Town/Cape_Town_22]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] was archived as [[etherpadbackup:p/Mach_mit!/2._Termin_Mentorinnen-Netzwerk]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] was archived as [[etherpadbackup:p/Kopdar_Wikisource_(Wikipabukon)_06/08/23]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] was archived as [[etherpadbackup:p/Igbo_Wikimedians_Community/Movement_Charter_Commun]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_C]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_C]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_B]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_B]] by [[User:BryanDavis|BryanDavis]] * 03:42 [[etherpad:p/GLAM-Treffen_2020/Gruppe_A]] was archived as [[etherpadbackup:p/GLAM-Treffen_2020/Gruppe_A]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] was archived as [[etherpadbackup:p/CEE_Talks/Update_CEE_Hub_/_Outlook_CEE_Meeting_202]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] was archived as [[etherpadbackup:p/CEE_Talks/Outlook_CEE_Meeting_2022_V2]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Wikispore]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Wikispore]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Translatable_modules]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Translatable_modules]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Opportunities_for_1Lib1Ref]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Movement_Strategy]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Movement_Strategy]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Lua_templates_for_beginners]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Lua_templates_for_beginners]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/Help_your_community_grow]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/Help_your_community_grow]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/CEE_Meeting_2020/101_Ways_to_Contribute]] was archived as [[etherpadbackup:p/CEE_Meeting_2020/101_Ways_to_Contribute]] by [[User:BryanDavis|BryanDavis]] * 03:40 [[etherpad:p/AsWikiMentoringProject2024/Oct-Dec]] was archived as [[etherpadbackup:p/AsWikiMentoringProject2024/Oct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:16 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:15 [[etherpad:p/AsWikiMentoringProject2024%2FOct-Dec]] was deleted as [[etherpadbackup:p/AsWikiMentoringProject2024%2FOct-Dec]] by [[User:BryanDavis|BryanDavis]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:wmh2023-Overview_of_technical_areas_%26_projects|wmh2023-Overview_of_technical_areas_%26_projects]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_20%2F04|open_meeting_COVID-19_20%2F04]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_18%2F04|open_meeting_COVID-19_18%2F04]] * 03:12 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:open_meeting_COVID-19_04%2F05|open_meeting_COVID-19_04%2F05]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:aswikimeet%2Fjuly-2024|aswikimeet%2Fjuly-2024]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikisource_Meetup_29%2F09%2F2020|Wikisource_Meetup_29%2F09%2F2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020|Wikimedians_of_Nepal%2FEvents%2FMeetup_February_2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Wikimedia_LGBT+%2FGovernance%2F2023-10-13|Wikimedia_LGBT+%2FGovernance%2F2023-10-13]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopD|WikiCendekia2026%2FWarkopD]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopC|WikiCendekia2026%2FWarkopC]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopB|WikiCendekia2026%2FWarkopB]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:WikiCendekia2026%2FWarkopA|WikiCendekia2026%2FWarkopA]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Summary%2F_Hypothesis|Summary%2F_Hypothesis]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(12%2F2024)|Stolpersteine_nach_WikiData_(12%2F2024)]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Stolpersteine_nach_WikiData_(01%2F2025|Stolpersteine_nach_WikiData_(01%2F2025]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Sommer_24%2F8,_1._August|Sommer_24%2F8,_1._August]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala|SVG_Translation_Campaign_2019_India%2FEvent%2FPatiala]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Meetup%2FCape_Town%2FCape_Town_22|Meetup%2FCape_Town%2FCape_Town_22]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk|Mach_mit!%2F2._Termin_Mentorinnen-Netzwerk]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23|Kopdar_Wikisource_(Wikipabukon)_06%2F08%2F23]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Igbo_Wikimedians_Community%2FMovement_Charter_Commun|Igbo_Wikimedians_Community%2FMovement_Charter_Commun]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_C|GLAM-Treffen_2020%2FGruppe_C]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_B|GLAM-Treffen_2020%2FGruppe_B]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:GLAM-Treffen_2020%2FGruppe_A|GLAM-Treffen_2020%2FGruppe_A]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:Commons_Photographers_30%2F05%2F2020|Commons_Photographers_30%2F05%2F2020]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:ChristianSW%2FNotizen|ChristianSW%2FNotizen]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202|CEE_Talks%2FUpdate_CEE_Hub_%2F_Outlook_CEE_Meeting_202]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Talks%2FOutlook_CEE_Meeting_2022_V2|CEE_Talks%2FOutlook_CEE_Meeting_2022_V2]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FWikispore|CEE_Meeting_2020%2FWikispore]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FTranslatable_modules|CEE_Meeting_2020%2FTranslatable_modules]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref|CEE_Meeting_2020%2FOpportunities_for_1Lib1Ref]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FMovement_Strategy|CEE_Meeting_2020%2FMovement_Strategy]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FLua_templates_for_beginners|CEE_Meeting_2020%2FLua_templates_for_beginners]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2FHelp_your_community_grow|CEE_Meeting_2020%2FHelp_your_community_grow]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:CEE_Meeting_2020%2F101_Ways_to_Contribute|CEE_Meeting_2020%2F101_Ways_to_Contribute]] * 03:11 [[etherpad:p/BryanDavis]] was deleted as [[etherpadbackup:p/BryanDavis]] by [[User:AsWikiMentoringProject2024%2FOct-Dec|AsWikiMentoringProject2024%2FOct-Dec]] * 01:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:51 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:50 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:49 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:44 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:43 [[etherpad:p/foo]] was archived as [[etherpadbackup:p/foo]] by [[User:EtherpadBackupBot|EtherpadBackupBot]] * 00:05 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] === 2026-04-26 === * 23:25 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 21:58 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:51 [[etherpad:p/bd808_test/path/?query=param]] was archived as [[etherpadbackup:p/bd808_test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:22 [[etherpad:p/bd808 test/path/?query=param]] was archived as [[etherpadbackup:p/bd808 test/path/?query=param]] by [[User:BryanDavis|BryanDavis]] * 17:22 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 16:17 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] === 2026-04-20 === * 02:42 [[etherpad:p/bd808-etherpad-backup-test]] was archived as [[etherpadbackup:p/bd808-etherpad-backup-test]] by [[User:BryanDavis|BryanDavis]] * 02:36 [[etherpad:p/bd808-etherpad-backup-test]] was archived by [[User:BryanDavis]] a0buxq57bhccmyvbik79r8uti0t72vc